test binary_data - error: local variable 'npz_handle' referenced before assignment
Hi Uwe,
I tested the binary data functionality today. There're a few issues.
Regarding the packing functionalities of the poolkit, that needs some tidying up, specifically:
binary_data_packing.py
- pillow missing as dependency.
- no documentation of packing and unpacking
- packing and unpacking images not used what are these functions for?
- test not with tests.
Regarding the actual digesting of the binary data. I used this code to play around:
from poolkit.binary_data_packing import pack, unpack
from poolkit.yaml_parser import print_yaml
# create fake data
import numpy as np
fake_data_arrays = [np.repeat(1, 8).reshape(2, 2, 2)]
# pack data and save it as binary
packed = pack(fake_data_arrays)
with open("/data2/landing_zone/data/sensor_from_xyz/sensor_abc_from_xyz/binary_data/test_file.npz", "wb") as binary:
binary.write(packed)
with open("/data2/landing_zone/data/sensor_from_xyz/sensor_abc_from_xyz/binary_data/binary_data_spec.yaml", "w") as yaml:
data = {
"site": "test_site",
"parameter": "images",
"timestamp": "2020-12-22 13:30:00",
"file": "test_file.npz"
}
print_yaml(data, fh=yaml)
When writing the yaml file first I get:
May 31 14:33:27 jasmin docker-compose[1381]: docker-datapool-1 | - dispatch data/sensor_from_xyz/sensor_abc_from_xyz/binary_data/binary_data_spec.yaml
May 31 14:33:27 jasmin docker-compose[1381]: docker-datapool-1 | error: /lz/data/sensor_from_xyz/sensor_abc_from_xyz/binary_data/test_file.npz might be corrupt: Cannot load file con>
May 31 14:33:27 jasmin docker-compose[1381]: docker-datapool-1 | error: local variable 'npz_handle' referenced before assignment
May 31 14:33:27 jasmin docker-compose[1381]: docker-datapool-1 | dispatch done
May 31 14:33:27 jasmin docker-compose[1381]: docker-datapool-1 | - dispatch data/sensor_from_xyz/sensor_abc_from_xyz/binary_data/test_file.npz
May 31 14:33:27 jasmin docker-compose[1381]: docker-datapool-1 | skip data/sensor_from_xyz/sensor_abc_from_xyz/binary_data/test_file.npz
May 31 14:33:27 jasmin docker-compose[1381]: docker-datapool-1 | dispatch done
When writing the binary_data first:
May 31 14:32:41 jasmin docker-compose[1381]: docker-datapool-1 | - dispatch data/sensor_from_xyz/sensor_abc_from_xyz/binary_data/test_file.npz
May 31 14:32:41 jasmin docker-compose[1381]: docker-datapool-1 | skip data/sensor_from_xyz/sensor_abc_from_xyz/binary_data/test_file.npz
May 31 14:32:41 jasmin docker-compose[1381]: docker-datapool-1 | dispatch done
May 31 14:32:41 jasmin docker-compose[1381]: docker-datapool-1 | - dispatch data/sensor_from_xyz/sensor_abc_from_xyz/binary_data/binary_data_spec.yaml
May 31 14:32:41 jasmin docker-compose[1381]: docker-datapool-1 | error: /lz/data/sensor_from_xyz/sensor_abc_from_xyz/binary_data/test_file.npz might be corrupt: Cannot load file con>
May 31 14:32:41 jasmin docker-compose[1381]: docker-datapool-1 | error: local variable 'npz_handle' referenced before assignment
May 31 14:32:41 jasmin docker-compose[1381]: docker-datapool-1 | dispatch done
Which one should be written first, yml or npz?
Is the data moved to the backup landing zone, or does it stay in the lz?
Have a great one! Christian
Edited by Christian Foerster