To save a file in the storage layer, a developer can call the putFile function passing in the file like so:
storage.putFile(filePath). This is a function call that will return the unique file id of the file originally located at the
Firstly we can outline this using high level state machine diagrams for
ProviderLink as follows.
NOTE: Yellow boxes represent communication to the linked node.
Storage.putFile- the public storage interface for storing a file on the network. It calls
Storage.enqueueFileForUploadand waits to get the file id of the file (file may not be uploaded yet). Then waits until the File uploading status is
File.UPLOADING_STATUS_UPLOADED. It also calls
Storage.chunkUploadingTickto ensure the chunks are uploaded based on the expectations set in the file metadata.
Storage.enqueueFileForUpload- This calls
File.chunkify(see details below) and sets the files metadata for redundancy, expiry and auto renew policies. Once the file is broken into chunks it calls
File.chunkify- splits the file by breaking the file into chunks of equal size of bytes upto the last chunk which may be smaller in size. It also creates a merkle tree using the chunk ids as the leaves. Each chunk has an initial state of
File.reconsiderUploadingStatus- sets the chunks metadata for redundancy, expiry and auto renew policies and then calls
Chunk.reconsiderUploadingStatusfor each chunk (see below).
Chunk.reconsiderUploadingStatus- checks that the chunk has the expected data redundancy (i.e. that it is stored in at least N providers) and that it has not expired. If any of these are not settled then the status of the chunk is set to Chunk.
UPLOADING_STATUS_UPLOADINGotherwise it is considered uploaded and so the status is set to
Storage.chunkUploadingTick- this function is called as a result of the original call to
Storage.putFileand it is also called on every ‘tick’ if there are any chunks that have a status Chunk.
UPLOADING_STATUS_UPLOADING. In this function the chunks Storage Links are pushed through various states of the uploading process and we can run though these states in the next step.
Storage.chunkUploadingTick function checks, and potentially, modifies the state of all the Storage Links of an uploading file chunk. The state of the chunk must be in
Chunk.UPLOADING_STATUS_UPLOADING for this function to proceed.
The function proceeds as follows:
Checks if the uploading chunk is required to be pushed to additional storage providers. It does this by fetching all the Storage Links for the chunk that have not failed and then compares that with the chunk redundancy to determine if more are required.
StorageLink.STATUS_CREATED: If additional Storage Providers are required then one is chosen using
chooseProviderCandidate()function. The process of choosing a provider candidate is detailed elsewhere in the docs. A new Storage Link is created and the provider, redkeyId, and chunk id is set on this new instance.
STORE_CHUNK_REQUESTcall is made to the linked Storage Provider and if agreed a check to determine if there is an existing payment channel between the client and the storage provider is carried out using
checkExistingChannel()function. If a channel exists the status is updated to
StorageLink.STATUS_AGREEDotherwise it is updated to
STORE_CHUNK_REQUESTcall fails it is updated to
STORE_CHUNK_REQUESTcall is handled using a custom Kademlia plugin or middleware that defines this handler (more on networking in a separate section of the documentation). Note this is a request to another node to store the chunk and at this point the actual chunk data is not sent to the other node - only the id, length and expiry parameters are sent.
StorageLink.STATUS_ESTABLISH_PAYMENT_CHANNEL: The function then checks in the
StorageLink.STATUS_ESTABLISH_PAYMENT_CHANNELstate and creates a payment channel with the Storage Provider using
createChannel()function. if successful the status is updated to
StorageLink.STATUS_ENCRYPTING: The function then checks all the Storage Links in the
StorageLink.STATUS_AGREEDstate and begins the encryption process. The Storage Link state is first updated to
StorageLink.STATUS_ENCRYPTINGand then the encryption process is performed by a forked child process that returns a message to the parent process when the encryption is complete.
StorageLink.STATUS_ENCRYPTED: The resulting encrypted chunk is split into multiple segment hashes used to produce a Merkle Tree representation of the chunk. This is then used to update the Storage Link once more with the resulting encrypted hash, encryption data length, segment hashes, merkle tree and merkle tree root. The status is updated to
StorageLink.STATUS_ENCRYPTEDat this point or
StorageLink.STATUS_FAILEDif the encryption process failed.
StorageLink.STATUS_SENDING_SEGMENT_MAP: The function then checks all the Storage Links in the
StorageLink.STATUS_ENCRYPTEDstate and begins sending the segment map to the linked Storage Provider. It first sets the state to
StorageLink.STATUS_SENDING_SEGMENT_MAPand then calls
STORE_CHUNK_SEGMENTSon the linked Storage Provider. The status is then updated to either
StorageLink.STATUS_SENDING_DATAif the request was successful or
StorageLink.STATUS_DATA_RECEIVEDif the response was ECHUNKALREADYSTORED from the linked Storage Provider. As always, if there was an error then the status is set to
StorageLink.STATUS_SENDING_DATA: The function then checks all the Storage Links in the
StorageLink.STATUS_SENDING_DATAstate and then prepares the data to be sent to the linked Storage Provider (which includes checking if chunks should be retransmitted due to a timeout condition). Once the data is prepared it calls
STORE_CHUNK_DATAwhich will invoke the equivalent function on the linked Storage Provider instance which will then store the data sent. On successful transmission of the data the Storage Link status is updated to
StorageLink.STATUS_DATA_RECEIVEDotherwise it is updated to
StorageLink.STATUS_DATA_RECEIVED: The function then checks all the Storage Links in the
StorageLink.STATUS_DATA_RECEIVEDstate and updates the state to
StorageLink.STATUS_ASKING_FOR_SIGNATUREand then sends a request to the linked Storage Provider for it to sign the chunk. Once it is successfully signed the status is changed to
StorageLink.STATUS_SIGNEDand the agreed micropayment is made to the Storage Provider using the makePayment() function. otherwise it is updated to