Creating an IPFS UnixFS File System By Hand

I am contemplating a collaborative file system based on IPFS. Users will publish the root of a directory tree & a service worker will copy that tree into a Neo4j instance running in Docker with a model of a versioned repository in it.

The frontend, then, will talk to Neo4j (which is a graph database) to allow more complex views of the data than a simple tree view.

I'm particularly interested in incorporating ordered lists where users rank content. Say I could get 50 people's opinions of the top 20 funniest memes ever, & generate a composite list from those.

To avoid having to upload a ton of information for every update, I want to reuse the unchanged parts of the previous iteration that are already in IPFS rather than reupload.

To do this, I need to manually construct the directory tree. I had it working with a CBOR-DAG, but the

ipfs dag

ipfs dag

commands are only available on instances where you have API access. So, I want to switch to UnixFS to store stuff.

I've tried to get Claude, V0, Co-Pilot, & Bolt to help me, and this is where we're at currently:

#!/usr/bin/env node

import { create as ipfsClient } from 'kubo-rpc-client'
import { prepare, encode } from '@ipld/dag-pb'
import { UnixFS } from 'ipfs-unixfs'

const ipfs = ipfsClient('http://localhost:5001')

try {
  const content = '¡Hello, World!'
  const file = new UnixFS({
    type: 'file',
    data: (
      typeof content === 'string' ? (new TextEncoder().encode(content)) : (content)
    ),
  })
  const fileNode = prepare({ Data: file.marshal(), Links: [] })
  const fileBytes = encode(fileNode)
  const start = await ipfs.add(fileBytes, { layout: 'dag-pb' })
  console.info(`Start CID: ${start.cid.toString()}`)

  const dir = new UnixFS({ type: 'directory' })
  const dirNode = (
    prepare({
      Data: dir.marshal(),
      Links: [{ Hash: start.cid, Name: 'hello.txt', Tsize: file.fileSize() }],
    })
  )
  const dirBytes = encode(dirNode)
  const final  = await ipfs.add(dirBytes, { layout: 'dag-pb' })
  console.info(`Final CID: ${final.cid.toString()}`)
} catch(error) {
  console.error(`Error: ${error.message}`)
  console.error(error.stack)
}

#!/usr/bin/env node

import { create as ipfsClient } from 'kubo-rpc-client'
import { prepare, encode } from '@ipld/dag-pb'
import { UnixFS } from 'ipfs-unixfs'

const ipfs = ipfsClient('http://localhost:5001')

try {
  const content = '¡Hello, World!'
  const file = new UnixFS({
    type: 'file',
    data: (
      typeof content === 'string' ? (new TextEncoder().encode(content)) : (content)
    ),
  })
  const fileNode = prepare({ Data: file.marshal(), Links: [] })
  const fileBytes = encode(fileNode)
  const start = await ipfs.add(fileBytes, { layout: 'dag-pb' })
  console.info(`Start CID: ${start.cid.toString()}`)

  const dir = new UnixFS({ type: 'directory' })
  const dirNode = (
    prepare({
      Data: dir.marshal(),
      Links: [{ Hash: start.cid, Name: 'hello.txt', Tsize: file.fileSize() }],
    })
  )
  const dirBytes = encode(dirNode)
  const final  = await ipfs.add(dirBytes, { layout: 'dag-pb' })
  console.info(`Final CID: ${final.cid.toString()}`)
} catch(error) {
  console.error(`Error: ${error.message}`)
  console.error(error.stack)
}

The output I get from that is:

Start CID: Qmc4KqktSNtJimrTJB5aKujCD27P7sV2AH48CeUbJZ4yJm
Final CID: QmcZA3Zohp3UtsPafQfSg4kPPpxZtGWMeQJvW7dW9WbyLx
❯ ipfs dag get Qmc4KqktSNtJimrTJB5aKujCD27P7sV2AH48CeUbJZ4yJm
{"Data":{"/":{"bytes":"CAISFwoVCAISD8KhSGVsbG8sIFdvcmxkIRgPGBc"}},"Links":[]}
❯ ipfs dag get QmcZA3Zohp3UtsPafQfSg4kPPpxZtGWMeQJvW7dW9WbyLx
{"Data":{"/":{"bytes":"CAISNRIvCiISIMvU/WQImfeyfmFxszzo6JKoCnjTRTSoY4rD1esrAmr+EgloZWxsby50eHQKAggBGDU"}},"Links":[]}
❯ ipfs cat Qmc4KqktSNtJimrTJB5aKujCD27P7sV2AH48CeUbJZ4yJm | cat -A
$
^U^H^B^R^OM-BM-!Hello, World!^X^O
❯ ipfs cat QmcZA3Zohp3UtsPafQfSg4kPPpxZtGWMeQJvW7dW9WbyLx | cat -A
^R/$
"^R M-KM-TM-}d^HM-^YM-wM-2~aqM-3<M-hM-hM-^RM-($
xM-SE4M-(cM-^JM-CM-UM-k+^BjM-~^R^Ihello.txt$
^B^H^A

Start CID: Qmc4KqktSNtJimrTJB5aKujCD27P7sV2AH48CeUbJZ4yJm
Final CID: QmcZA3Zohp3UtsPafQfSg4kPPpxZtGWMeQJvW7dW9WbyLx
❯ ipfs dag get Qmc4KqktSNtJimrTJB5aKujCD27P7sV2AH48CeUbJZ4yJm
{"Data":{"/":{"bytes":"CAISFwoVCAISD8KhSGVsbG8sIFdvcmxkIRgPGBc"}},"Links":[]}
❯ ipfs dag get QmcZA3Zohp3UtsPafQfSg4kPPpxZtGWMeQJvW7dW9WbyLx
{"Data":{"/":{"bytes":"CAISNRIvCiISIMvU/WQImfeyfmFxszzo6JKoCnjTRTSoY4rD1esrAmr+EgloZWxsby50eHQKAggBGDU"}},"Links":[]}
❯ ipfs cat Qmc4KqktSNtJimrTJB5aKujCD27P7sV2AH48CeUbJZ4yJm | cat -A
$
^U^H^B^R^OM-BM-!Hello, World!^X^O
❯ ipfs cat QmcZA3Zohp3UtsPafQfSg4kPPpxZtGWMeQJvW7dW9WbyLx | cat -A
^R/$
"^R M-KM-TM-}d^HM-^YM-wM-2~aqM-3<M-hM-hM-^RM-($
xM-SE4M-(cM-^JM-CM-UM-k+^BjM-~^R^Ihello.txt$
^B^H^A

So, my file attempt has a bunch of random crap in with the data, & my directory doesn't have any children. When I open either up in the IPFS Desktop it says they're not browsable.

KP-C

Kevin Powell - Community•17mo ago

dys 🐙

Creating an IPFS UnixFS File System By Hand

ipfs dag

ipfs dag

#!/usr/bin/env node

import { create as ipfsClient } from 'kubo-rpc-client'
import { prepare, encode } from '@ipld/dag-pb'
import { UnixFS } from 'ipfs-unixfs'

const ipfs = ipfsClient('http://localhost:5001')

try {
  const content = '¡Hello, World!'
  const file = new UnixFS({
    type: 'file',
    data: (
      typeof content === 'string' ? (new TextEncoder().encode(content)) : (content)
    ),
  })
  const fileNode = prepare({ Data: file.marshal(), Links: [] })
  const fileBytes = encode(fileNode)
  const start = await ipfs.add(fileBytes, { layout: 'dag-pb' })
  console.info(`Start CID: ${start.cid.toString()}`)

  const dir = new UnixFS({ type: 'directory' })
  const dirNode = (
    prepare({
      Data: dir.marshal(),
      Links: [{ Hash: start.cid, Name: 'hello.txt', Tsize: file.fileSize() }],
    })
  )
  const dirBytes = encode(dirNode)
  const final  = await ipfs.add(dirBytes, { layout: 'dag-pb' })
  console.info(`Final CID: ${final.cid.toString()}`)
} catch(error) {
  console.error(`Error: ${error.message}`)
  console.error(error.stack)
}

#!/usr/bin/env node

import { create as ipfsClient } from 'kubo-rpc-client'
import { prepare, encode } from '@ipld/dag-pb'
import { UnixFS } from 'ipfs-unixfs'

const ipfs = ipfsClient('http://localhost:5001')

try {
  const content = '¡Hello, World!'
  const file = new UnixFS({
    type: 'file',
    data: (
      typeof content === 'string' ? (new TextEncoder().encode(content)) : (content)
    ),
  })
  const fileNode = prepare({ Data: file.marshal(), Links: [] })
  const fileBytes = encode(fileNode)
  const start = await ipfs.add(fileBytes, { layout: 'dag-pb' })
  console.info(`Start CID: ${start.cid.toString()}`)

  const dir = new UnixFS({ type: 'directory' })
  const dirNode = (
    prepare({
      Data: dir.marshal(),
      Links: [{ Hash: start.cid, Name: 'hello.txt', Tsize: file.fileSize() }],
    })
  )
  const dirBytes = encode(dirNode)
  const final  = await ipfs.add(dirBytes, { layout: 'dag-pb' })
  console.info(`Final CID: ${final.cid.toString()}`)
} catch(error) {
  console.error(`Error: ${error.message}`)
  console.error(error.stack)
}

The output I get from that is:

Start CID: Qmc4KqktSNtJimrTJB5aKujCD27P7sV2AH48CeUbJZ4yJm
Final CID: QmcZA3Zohp3UtsPafQfSg4kPPpxZtGWMeQJvW7dW9WbyLx
❯ ipfs dag get Qmc4KqktSNtJimrTJB5aKujCD27P7sV2AH48CeUbJZ4yJm
{"Data":{"/":{"bytes":"CAISFwoVCAISD8KhSGVsbG8sIFdvcmxkIRgPGBc"}},"Links":[]}
❯ ipfs dag get QmcZA3Zohp3UtsPafQfSg4kPPpxZtGWMeQJvW7dW9WbyLx
{"Data":{"/":{"bytes":"CAISNRIvCiISIMvU/WQImfeyfmFxszzo6JKoCnjTRTSoY4rD1esrAmr+EgloZWxsby50eHQKAggBGDU"}},"Links":[]}
❯ ipfs cat Qmc4KqktSNtJimrTJB5aKujCD27P7sV2AH48CeUbJZ4yJm | cat -A
$
^U^H^B^R^OM-BM-!Hello, World!^X^O
❯ ipfs cat QmcZA3Zohp3UtsPafQfSg4kPPpxZtGWMeQJvW7dW9WbyLx | cat -A
^R/$
"^R M-KM-TM-}d^HM-^YM-wM-2~aqM-3<M-hM-hM-^RM-($
xM-SE4M-(cM-^JM-CM-UM-k+^BjM-~^R^Ihello.txt$
^B^H^A

Start CID: Qmc4KqktSNtJimrTJB5aKujCD27P7sV2AH48CeUbJZ4yJm
Final CID: QmcZA3Zohp3UtsPafQfSg4kPPpxZtGWMeQJvW7dW9WbyLx
❯ ipfs dag get Qmc4KqktSNtJimrTJB5aKujCD27P7sV2AH48CeUbJZ4yJm
{"Data":{"/":{"bytes":"CAISFwoVCAISD8KhSGVsbG8sIFdvcmxkIRgPGBc"}},"Links":[]}
❯ ipfs dag get QmcZA3Zohp3UtsPafQfSg4kPPpxZtGWMeQJvW7dW9WbyLx
{"Data":{"/":{"bytes":"CAISNRIvCiISIMvU/WQImfeyfmFxszzo6JKoCnjTRTSoY4rD1esrAmr+EgloZWxsby50eHQKAggBGDU"}},"Links":[]}
❯ ipfs cat Qmc4KqktSNtJimrTJB5aKujCD27P7sV2AH48CeUbJZ4yJm | cat -A
$
^U^H^B^R^OM-BM-!Hello, World!^X^O
❯ ipfs cat QmcZA3Zohp3UtsPafQfSg4kPPpxZtGWMeQJvW7dW9WbyLx | cat -A
^R/$
"^R M-KM-TM-}d^HM-^YM-wM-2~aqM-3<M-hM-hM-^RM-($
xM-SE4M-(cM-^JM-CM-UM-k+^BjM-~^R^Ihello.txt$
^B^H^A

So, my file attempt has a bunch of random crap in with the data, & my directory doesn't have any children. When I open either up in the IPFS Desktop it says they're not browsable.

Creating an IPFS UnixFS File System By Hand

Creating an IPFS UnixFS File System By Hand

Similar Threads

Similar Threads

Similar Threads