Konubinix' opinionated web of thoughts

Ipfs

Fleeting

A peer-to-peer file system where each node stores its own data and the network maintains a DHT to be able to ask the correct node when wanting the get the data.

Thus, wherever you are, you can get the data and ipfs will ask the dht about the node that possesses the data so as to get it for you.

It is that incredible that I eventually decide to organize all my static files with it, using org-roam.

check if cid available in the node offline

ipfs refs --offline {cid}

Or, more manually, follow the part from ipfs cid to block storage and check that all the block are there.

from ipfs cid to block storage

cat ~/.ipfs/blocks/_README

This is a repository of IPLD objects. Each IPLD object is in a single file,
named <base32 encoding of cid>.data. Where <base32 encoding of cid> is the
"base32" encoding of the CID (as specified in
https://github.com/multiformats/multibase) without the 'B' prefix.
All the object files are placed in a tree of directories, based on a
function of the CID. This is a form of sharding similar to
the objects directory in git repositories. Previously, we used
prefixes, we now use the next-to-last two characters.

    func NextToLast(base32cid string) {
      nextToLastLen := 2
      offset := len(base32cid) - nextToLastLen - 1
      return str[offset : offset+nextToLastLen]
    }

For example, an object with a base58 CIDv1 of

    zb2rhYSxw4ZjuzgCnWSt19Q94ERaeFhu9uSqRgjSdx9bsgM6f

has a base32 CIDv1 of

    BAFKREIA22FLID5AJ2KU7URG47MDLROZIH6YF2KALU2PWEFPVI37YLKRSCA

and will be placed at

    SC/AFKREIA22FLID5AJ2KU7URG47MDLROZIH6YF2KALU2PWEFPVI37YLKRSCA.data

with 'SC' being the last-to-next two characters and the 'B' at the
beginning of the CIDv1 string is the multibase prefix that is not
stored in the filename.

I can add a block in the datastore that way.

content="$(d=$(date -Ins);for i in {1..10000}; do echo $d;done)"
find ~/.ipfs/blocks -type f > /tmp/before
cid=$(echo -n "${content}"| ipfs block put)
echo ${cid}
find ~/.ipfs/blocks -type f > /tmp/after
new_line=$(konix_diff_new_lines.sh /tmp/before /tmp/after)
echo "# Lines added in the data store"
echo "${new_line}"
test "$(cat "${new_line}")" = "${content}" && echo "This file contains the content"
echo "# What ipfs cid format shows"
ipfs cid format -b base32upper -f "\
  multibase name                                                      %b
  multibase code                                                      %B
  version string                                                      %v
  version number                                                      %V
  codec name                                                          %c
  codec code                                                          %C
  multihash name                                                      %h
  multihash code                                                      %H
  hash digest length                                                  %L
  multihash encoded in base %%b (with multibase prefix)                %m
  multihash encoded in base %%b without multibase prefix               %M
  hash digest encoded in base %%b (with multibase prefix)              %d
  hash digest encoded in base %%b without multibase prefix             %D
  cid string encoded in base %%b (1)                                   %s
  cid string encoded in base %%b without multibase prefix              %S
" "${cid}"

bafkreicidn56flusss4rpp2inmom2a7lzc3j3puec5sp5fpohh5wmkeeby
# Lines added in the data store
/home/sam/.ipfs/blocks/ID/CIQEQG334KXJFFFZC67UQ2Y4ZUB6XSFWTW7IIF3E72K64OP3MYUIIDQ.data
This file contains the content
# What ipfs cid format shows
  multibase name                                                      base32upper
  multibase code                                                      B
  version string                                                      cidv1
  version number                                                      1
  codec name                                                          raw
  codec code                                                          85
  multihash name                                                      sha2-256
  multihash code                                                      18
  hash digest length                                                  32
  multihash encoded in base %b (with multibase prefix)                BCIQEQG334KXJFFFZC67UQ2Y4ZUB6XSFWTW7IIF3E72K64OP3MYUIIDQ
  multihash encoded in base %b without multibase prefix               CIQEQG334KXJFFFZC67UQ2Y4ZUB6XSFWTW7IIF3E72K64OP3MYUIIDQ
  hash digest encoded in base %b (with multibase prefix)              BJANXXYVOSKKLSF57JBVRZTID5PELNHN6QQLWJ7UV5Y47WZRIQQHA
  hash digest encoded in base %b without multibase prefix             JANXXYVOSKKLSF57JBVRZTID5PELNHN6QQLWJ7UV5Y47WZRIQQHA
  cid string encoded in base %b (1)                                   BAFKREICIDN56FLUSSS4RPP2INMOM2A7LZC3J3PUEC5SP5FPOHH5WMKEEBY
  cid string encoded in base %b without multibase prefix              AFKREICIDN56FLUSSS4RPP2INMOM2A7LZC3J3PUEC5SP5FPOHH5WMKEEBY

If I try to create a content bigger than 1MiB, ipfs complains and suggests using unixfs to chunk the data instead.

I can see that using the ‘%M’ formatted, I can find the associated multihash that is used to find the path in the filesystem.

Now, I can add a big file using ipfs add and see how the chunks are put into blocks.

content="$(d=$(date -Ins);for i in {1..10000}; do echo $d;done)"
find ~/.ipfs/blocks -type f > /tmp/before
cid=$(echo -n "${content}"|ipfa)
cid="${cid#/ipfs/*}"
echo "Cid of my content: ${cid}"
find ~/.ipfs/blocks -type f > /tmp/after
files=$(konix_diff_new_lines.sh /tmp/before /tmp/after)
echo "# files added by the ipfs add command"
echo "${files}"

ipfs dag get "${cid}" && echo
cids=("${cid}" $(ipfs refs -r "${cid}"))

extra_files="${files}"
for part in "${cids[@]}"
do
    hash=$(ipfs cid format -b base32upper -f "%M" "${part}")
    echo "${cid} stored in block of hash ${hash}"
    extra_files="$(echo "${extra_files}"|grep -v "${hash}")"
done

echo "${extra_files}"
rm "${extra_files}"

test "$(ipfs cat "${cid}")" = "${content}" && echo "Remove the extra file and can still retrieve the content"

Cid of my content: bafybeididvnmde2xczrwhyhauimbfsfucledmmsnakt7wrb7ecfxpoxfl4
# files added by the command
/home/sam/.ipfs/blocks/KX/CIQGQHK2YGJVOFTDMPQOBIQYCLELIEWIGYZE2AVH7NCD6IELO65OKXY.data
/home/sam/.ipfs/blocks/AV/CIQMUE2AVU6BT2ZHX4SQPCVVZ3B4VYOAIHSBGRRIWFDQ6SXWBTALAVQ.data
/home/sam/.ipfs/blocks/7L/CIQGYU63VU2EQEQLZBYQ5ZVWJZ3YX6NDL5MCZU6MKZ4VNKEWG37V7LA.data
/home/sam/.ipfs/blocks/FJ/CIQJOCYSQPYBPJZ6OVDVMV427YVYZOL37V3XDZSZ4XOZSXB3F62ZFJQ.data
{"Data":{"/":{"bytes":"CAIYv/wVIICAECC//AU"}},"Links":[{"Hash":{"/":"bafkreigkcnak2paz5mt36jihrk245q6k4haedzatiyulcrypjl3azqfqky"},"Name":"","Tsize":262144},{"Hash":{"/":"bafkreiexbmjih4axu47hkr2wk6np4k4mxf57253r4zm6lxmzlq5s7nmsuy"},"Name":"","Tsize":97855}]}
bafybeididvnmde2xczrwhyhauimbfsfucledmmsnakt7wrb7ecfxpoxfl4 stored in block of hash CIQGQHK2YGJVOFTDMPQOBIQYCLELIEWIGYZE2AVH7NCD6IELO65OKXY
bafybeididvnmde2xczrwhyhauimbfsfucledmmsnakt7wrb7ecfxpoxfl4 stored in block of hash CIQMUE2AVU6BT2ZHX4SQPCVVZ3B4VYOAIHSBGRRIWFDQ6SXWBTALAVQ
bafybeididvnmde2xczrwhyhauimbfsfucledmmsnakt7wrb7ecfxpoxfl4 stored in block of hash CIQJOCYSQPYBPJZ6OVDVMV427YVYZOL37V3XDZSZ4XOZSXB3F62ZFJQ
/home/sam/.ipfs/blocks/7L/CIQGYU63VU2EQEQLZBYQ5ZVWJZ3YX6NDL5MCZU6MKZ4VNKEWG37V7LA.data
Remove the extra file and can still retrieve the content

We can see that the command created a main cid and two children one. Yet it created 4 files instead of 3 as I expected. By computing the hash of the 3 expected files, I can remove the extra one and find out that ipfs still behaves as intended. I suppose this is a side effect of the ipfs add command.

Therefore, I suppose I can find the relevant blocks using the following script.

cids=("${cid}" $(ipfs refs -r "${cid}"))

extra_files="${files}"
for part in "${cids[@]}"
do
    hash=$(ipfs cid format -b base32upper -f "%M" "${part}")
    echo "${IPFS_PATH-${HOME}/.ipfs}/blocks/${hash: -3:2}/${hash}.data"
done

/home/sam/.ipfs/blocks/KX/CIQGQHK2YGJVOFTDMPQOBIQYCLELIEWIGYZE2AVH7NCD6IELO65OKXY.data
/home/sam/.ipfs/blocks/AV/CIQMUE2AVU6BT2ZHX4SQPCVVZ3B4VYOAIHSBGRRIWFDQ6SXWBTALAVQ.data
/home/sam/.ipfs/blocks/FJ/CIQJOCYSQPYBPJZ6OVDVMV427YVYZOL37V3XDZSZ4XOZSXB3F62ZFJQ.data

ipfs how to know the block number of a cid

ipfs dag stat CID

Mutable File System

MFS acts as a single, dynamic filesystem mount. MFS has a root CID that is transparently updated when a change happens (and can be checked with “ipfs files stat /”).

ipfs files –help

Notes linking here