At the moment our storage node is comprised of two parts -
- Authentication layer:
- This checks to ensure that the user executing the file related operation has the required permissions to do so
- A S3-compatible object store
Whenever a user wants to perform an operation like file upload they first connect to the gateway node using the Arcana SDK integrated in their APP and get their request validated. Once validated, the gateway node provides the user with a time bounded token & a storage node endpoint to connect to in order to perform the upload. The user can then send the file to be uploaded to the storage node endpoint after the token and permissions have been validated by the authentication layer.
Arcana uses MinIO object stores with erasure coding at the moment to store data. Erasure coding is a way to reconstruct missing or corrupted data in order to protect it against hardware failures and data corruption. Specifically, MinIO uses Reed-Solomon code to shard objects into data and parity blocks (default is data blocks & parity blocks). Here one can lose blocks and still be able to reconstruct the data completely. MinIO also uses HighwayHash checksums to ensure that the backend data is protected from data rot or silent data corruption. Data rot occurs when the data on the drive gets corrupted silently without signaling an error.