pbillingsby

pbillingsby

Decentralized storage with Arweave & Bundlr

pbillingsby.eth's photo
pbillingsby.eth
·May 22, 2022·

14 min read

The recent removal and swift return of the Bankless YouTube channel raised much-needed awareness (and eyebrows) in the Web3 community about using and relying on a centralized source to host your content.

People started conversations about the need for a Web3 YouTube and scrambled to find the right tech stack to make this work. I eagerly awaited suggestions and ideas on how to solve this problem; one thing was evident to me, Arweave should be an integral part of that solution.

Relying on a single source for storing data is complicated. It means there is a single point of failure. Data stored on centralized networks are less reliable for many reasons. It also introduces the problem of unexpected expenses, should that host decide to add charges on top of the agreed price. There is also the risk of revocation.

With Arweave, you don't have to worry about your data being held hostage due to unforeseen, additional costs to storing your data. With the Arweave storage model, you pay once and can trust that Arweave retains your data because it's backed by a storage endowment. This article will introduce decentralized storage, the Permaweb, and Bundlr, a scaling solution for Arweave that allows us to pay for storage with tokens other than Arweave's token $AR.

We will build an application that will allow us to upload an image file to Arweave. By the end of this, you will have a good idea of the capabilities of using decentralized storage and most of the tools needed to have your own personal decentralized data uploader.

Arweave

Arweave is a decentralized data storage protocol. It stores files permanently across a distributed network of computers, which makes up the Arweave network. It works similar to how Uber connects drivers to passengers; it connects people needing to store data with people that have l hard drive space to store that data.

Typical blockchains store chains of blocks that contain transactions. Arweave stores data in a graph of blocks. Each block links to two earlier blocks forming a structure called a blockweave.

The blockweave allows the network to force miners to provide a Proof of Access (PoA), which requires each miner to check each new bundle of transactions and also contains a randomly selected marker from a previous block.

On top of Arweave is a layer called Permaweb, a decentralized, immutable web. Although the Permaweb may seem like the regular web, everything deployed to it is permanent, offering a low-cost, zero-maintenance solution to hosting web applications and pages. Backing data with sustainable and perpetual endowments, meaning that our data is safely hosted, with no chance of additional fees once uploaded.

And these are only a few reasons why I believe in Arweave.

Check out their wiki!

Bundlr

Bundlr is a scaling technology for Arweave that increases the number of transactions by 4000% and upload speed by ~3000% without sacrificing security or usability, making building on top of Arweave more efficient, accessible, and affordable.

The AR token isn't easy to acquire, but with Bundlr, we can use ETH, MATIC, AVAX, and more currencies to pay for Arweave storage.

The Bundlr network comprises nodes, known as bundlers, which bundle multiple layer-2 transactions into a single Arweave layer-1 transaction. Each node will submit these transactions to the main Arweave network.

Once you submit a valid transaction to a bundler, you get a receipt as a financial commitment to seed the data on layer-1. Validators will then ensure the data is uploaded correctly by the block cutoff period defined in this receipt.

Bundlr is an important tool for working with Arweave as it not only makes it more affordable to use but is more accessible by introducing the use of different tokens. Find out more here


Getting Started

Now that we have had an overview on Arweave, Bundlr, and decentralized storage, we are going to be creating a decentralized image uploader. By the end of this you will have a better understanding of how to build on top of these tools and how simple it is to achieve decentralization of your data.

The first step is to change our MetaMask network to Polygon Mainnet. Then we want to create the Next.js app, change into its directory, and install the dependencies, which I will explain throughout the walkthrough.

npx create-next-app arweave-bundlr-app
cd arweave-bundlr-app
npm i @bundlr-network/client bignumber.js ethers

The next thing we want to do is open the project in our text editor. We will add code in both pages/_app.js and pages/index.js. As we are going to be making some of the functions and balances from _app.js available globally, we will need to create a context.js file at the root of the project, and in that file, we will create the main context

import { createContext } from 'react'
export const MainContext = createContext()

Now, we can start to write our code in _app.js, which will handle creating a Bundlr instance, and everything needed to load up with tokens and view the chosen token balance from that instance.

I will be using MATIC for this tutorial, but Bundlr, as stated above, can use many different tokens.

Initializing Bundlr Instance

Let's take a look at the code below briefly, copy and paste it into your _app.js file.

We add some generic styles to the code to make it easier when we view our app in the browser. I will not be going over this but feel free to add your own styles here.

Whatever funds you load your Bundlr instance with will be tied to your wallet, allowing us to use those funds in future developing when you connect that Bundlr intance.

A reminder: we will be using these functions and values in the index.js component, which we will dive into later.


Click to view code
import { MainContext } from '../context'
import { useState, useRef } from 'react'
import { providers, utils } from 'ethers'
import { WebBundlr } from '@bundlr-network/client'

function MyApp({ Component, pageProps }) {
  const [bundlrInst, setBundlrInst] = useState();
  const [balance, setBalance] = useState();
  const bundlrRef = useRef();

  const initialize = async () => {
    await window.ethereum.enable();
    const provider = new providers.Web3Provider(window.ethereum);
    await provider._ready();

    const bundlr = new WebBundlr('https://node1.bundlr.network', 'matic', provider)
    await bundlr.ready();

    setBundlrInst(bundlr);
    bundlrRef.current = bundlr;
    getBalance();
  }

  const getBalance = async () => {
    const bal = await bundlrRef.current.getLoadedBalance();
    setBalance(utils.formatEther(bal.toString()));
  }

  return (
    <div style={containerStyle}>
      <MainContext.Provider value={{
        initialize,
        getBalance,
        balance,
        bundlrInst
      }}>
        <h1>Arweave/Bundlr</h1>
        <Component {...pageProps} />
      </MainContext.Provider>
    </div>
  )
}
const containerStyle = {
  width: '80vw',
  margin: '40px auto',
  textAlign: 'center',
}
export default MyApp


First, we will import what we need from a few dependencies.

MainContext is the context we created before the code above. React hooks for state and ref. We will use providers to connect to MM and utils to format our balance. WebBundlr for creating our Bundlr instance and connecting to the network, which in our case is MATIC.

import { MainContext } from '../context'
import { useState, useRef } from 'react'
import { providers, utils } from 'ethers'
import { WebBundlr } from '@bundlr-network/client'

Next, nested inside the MyApp function, we need to declare a few variables before we move on to creating our Bundlr instance. We will get back to those soon when we write our functions.

const [bundlrInst, setBundlrInst] = useState();
const [balance, setBalance] = useState();
const bundlrRef = useRef();

Now, here is where the fun with Bundlr starts. We will initialize and set our instance, which then opens us up to being able to pay for Arweave transactions, and also will give us our balance in the token we specified when we created the instance.

The initialize function that we'll be triggering from the index.js file will connect to the Polygon network and set the provider we can pass in when connecting to the Bundlr network.

When we declare the bundlr variable, we pass three values. The first is the url to the Bundlr node we are connecting to, followed by the token we want to transact with, and then the provider we declared earlier in the code block. Once that promise resolves, we will use those values to set the state and ref, then retrieve the balance.

const initialize = async () => {
    await window.ethereum.enable();
    const provider = new providers.Web3Provider(window.ethereum);
    await provider._ready();

    const bundlr = new WebBundlr('https://node1.bundlr.network', 'matic', provider);
    await bundlr.ready();

    setBundlrInst(bundlr);
    bundlrRef.current = bundlr;
    getBalance();
}

Now, that we have the bundlrInst and bundlrRef set, the last thing we want to do before passing these to our next component is to set our balance variable with the token balance available. Here we need to format our balance.

Now, if we just took the value of bal below and displayed it, it's going to look something like this BigNumber {s: 1, e: 16, c: Array(2)}.

BigNumber is an object that safely allows mathematical operations on numbers of any magnitude.

bal.toString() will convert our BigNumber to a String, which will give us our balance in Wei, which will provide us with the smallest denomination of Ether (ETH).

We will then use the formatEther function we imported earlier from ethers to convert the Wei amount into the chosen token and set our balance state to the formatted balance.

  const getBalance = async () => {
    const bal = await bundlrRef.current.getLoadedBalance();
    setBalance(utils.formatEther(bal.toString()));
}

Then when we render our component, we wrap everything with our context, allowing us to pass the Bundlr values to the other components. We will only pass this to one component for this walkthrough, but you can imagine that having these values available will come in handy when you add more pages and components.

<MainContext.Provider value={{
  initialize,
  getBalance,
  balance,
  bundlrInst
}}>
  <h1>Arweave/Bundlr</h1>
  <Component {...pageProps} />
</MainContext.Provider>

Uploading to Arweave


Now it's time for us to take what we've done with Bundlr, create a UI with some buttons to load up our Bundlr instance with MATIC, and a file input that we will be persisting using Arweave. To run our local server, we want to run npm run dev in the terminal to start the server. Once running, you can navigate to your localhost://3000 on your browser.

Paste the code below into your index.js file, and we will break down this component below. Like before, there are some styles at the bottom for some of the elements, so feel free to play around and try and add your own flavor to this build.

Click to view code
import Head from 'next/head'
import { useState, useContext } from 'react'
import { MainContext } from '../context'
import BigNumber from 'bignumber.js'

export default function Home() {
  const [selectedFile, setSelectedFile] = useState();
  const [img, setImg] = useState();
  const [URI, setURI] = useState();
  const [amount, setAmount] = useState();

  const {
    initialize, getBalance, balance, bundlrInst
  } = useContext(MainContext);

  const uploadImage = async () => {
    let txn = await bundlrInst.uploader.upload(selectedFile, [{ name: "Content-Type", value: "image/png" }]);
    setURI(`http://arweave.net/${txn.data.id}`);
    getBalance();
  }

  const handleFileChange = (e) => {
    const reader = new FileReader();
    const file = e.target.files[0];
    if (file) {
      reader.onloadend = () => {
        if (reader.result) {
          setSelectedFile(Buffer.from(reader.result));
        }
      };
      reader.readAsArrayBuffer(file);
      const objectUrl = URL.createObjectURL(file);
      setImg(objectUrl);
    }
  }

  const fundWallet = async () => {
    if (!amount) return;
    const parseAmount = parseInput(amount);
    let response = await bundlrInst.fund(parseAmount);
    console.log('wallet funded: ', response);
    getBalance();
  }

  const parseInput = (input) => {
    const conversion = new BigNumber(input).multipliedBy(bundlrInst.currencyConfig.base[1]);
    if (conversion.isLessThan(1)) {
      console.log('error: value too small');
      return;
    } else {
      return conversion;
    }
  }

  return (
    <div>
      <Head>
        <title>Arweave/Bundlr</title>
        <meta name="description" content="Generated by create next app" />
        <link rel="icon" href="/favicon.ico" />
      </Head>

      <main>
        {!balance && <button onClick={initialize}>Initialize</button>}
        {balance && (
          <>
            <p>balance: {balance}</p>
            <input placeholder='add funds'
              onChange={(e) => setAmount(e.target.value)} />
            <button onClick={fundWallet}>fund</button>
          </>
        )
        }
        <div style={{ paddingTop: '10em' }}>
          <input
            type='file'
            onChange={handleFileChange}
          />
          <button style={button} onClick={uploadImage}>
            Upload Image
            </button>
        </div>
        {img &&
          <div style={previewFlex}>
            <div style={{ margin: '0 auto' }}>
              <h3>Our Image</h3>
              {img && (
                <div>
                  <img src={img} style={previewStyle} />
                </div>
              )
              }
            </div>
            <div style={{ margin: '0 auto' }}>
              <h3>Arweave Image</h3>
              {URI && (
                <>
                  <img src={URI} style={previewStyle} />
                  <a href={URI} target="_blank">{URI}</a>
                </>
              )
              }
            </div>
          </div>}
      </main>
    </div>
  )
}
const previewStyle = {
  maxWidth: '500px',
  height: 'auto',
  margin: '0 auto'
}

const previewFlex = {
  display: 'flex',
}

const button = {
  outline: 'none',
  border: '1px solid black',
  backgroundColor: 'white',
  padding: '10px',
  width: '200px',
  marginBottom: 10,
  cursor: 'pointer',
}


We will start to break this down, first looking at the state variables and how we will be using our context to get those values we passed into our parent component _app.js. Here we set a few variables to an empty state.

  • amount will be the amount we will be loading our wallet up within MATIC.
  • selectedFile is going to be set from the handleFileChange function when we select which file we want to upload, in this case, a PNG.
  • img URL will change after this happens, which we will use for our local preview of the image before uploading it to Arweave.

After we upload to Arweave, we will use that transaction data to set the URI, which will display our persisted image.

The last thing we're doing in this block is destructuring and assigning the values from the context value, making them available in this component.

const [amount, setAmount] = useState();
const [selectedFile, setSelectedFile] = useState();
const [img, setImg] = useState();
const [URI, setURI] = useState();

const {
  initialize, getBalance, balance, bundlrInst
} = useContext(MainContext);

Now we can see how the rest of these functions and elements interact to give us the result we need. First, we will get everything with Bundlr sorted out, starting with initializing our Bundlr instance from the UI.

Screen Shot 2022-05-22 at 8.46.34 AM.png

If you head to localhost, you will see how it looks so far. You will see a button nested at the top that says initialize on it. Once you click it, it will prompt you to sign a message from Bundlr. If you remember from the initialize function, we need to connect to Polygon Mainnet. Once initialized, you will see a text input to add funds. Let's take a look at how that works.

Adding Funds

{balance && (
  <>
    <p>balance: {balance}</p>
    <input placeholder='add funds' onChange={(e) => setAmount(e.target.value)} />
    <button onClick={fundWallet}>fund</button>
  </>
)}

Here we have a text input and button where the input has an onChange handler which will set the amount variable with the amount desired to load our Bundlr with. Once we click the button to fund, fundWallet will be triggered and pass the amount to the parseInput function to convert the input into a BigNumber using our imported library.

If the conversion is equal to less than 1, we will not be able to fund our Bundlr; if greater than 1, you have to approve the transaction. After receiving a successful response, getBalance will be called again to update our balance.

Now we have funds ready we can move on and get our image uploaded to Arweave.

const fundWallet = async () => {
  if (!amount) return;
  const parseAmount = parseInput(amount);
  let response = await bundlrInst.fund(parseAmount);
  console.log('wallet funded: ', response);
  getBalance();
}

const parseInput = (input) => {
  const conversion = new BigNumber(input).multipliedBy(bundlrInst.currencyConfig.base[1]);
  if (conversion.isLessThan(1)) {
    console.log('error: value too small');
    return
  } else {
    return conversion;
  }
}

Handling File Changes

We're all loaded up and ready to add our image to Arweave. Once your transaction is confirmed at the end of this, your image will live on the Arweave network for at least the following 200 years. We should be able to see our balance, along with the previous elements rendered. Upload your chosen image. You should now see a preview of the image rendered on the left, below Our Image.

Screen Shot 2022-05-22 at 9.17.13 AM.png

Once we get to the next step, our image will fill that space underneath Arweave Image and a link to its URI on the Arweave network. Now that we see our preview image, let's look at the handleChange function and how this prepares us for the final step of this walkthrough.

Here we create a new FileReader object and set the state of file to equal the image data from the file input. If you have selected your image, the function will wait for the reader load to end and then convert your file input into a blob, which will be our selectedFile which we persist to Arweave, and display underneath Arweave Image. After that, it will need to convert the blob and create a URL using URL.createObjectURL, to render our Our Image.

const handleFileChange = (e) => {
  const reader = new FileReader();
  const file = e.target.files[0];
  if (file) {
    reader.onloadend = () => {
      if (reader.result) {
        setSelectedFile(Buffer.from(reader.result));
      }
    };
    reader.readAsArrayBuffer(file);
    const objectUrl = URL.createObjectURL(file);
    setImg(objectUrl);
  }
}

Uploading Image to Arweave

The moment we've all been waiting for, we can now click the "Upload Image" button, approve the transaction, and be able to view our Arweave file and the transaction URI below it.

Once we upload the image, we pass the selectedFile, along with some headers to the Bundlr object which will handle uploading the image to Arweave, only for us to approve the transaction. Once that transaction has been created, we can retrieve data needed to create and set our URI by appending txn.data.id to the end of the Arweave network URL.

const uploadImage = async () => {
    let txn = await bundlrInst.uploader.upload(selectedFile, [{ name: "Content-Type", value: "image/png" }])
    setURI(`http://arweave.net/${txn.data.id}`);
  }

Once we've obtained that URI, we can use it to display our image and URI.

We did it!

Now you should see a display similar to the one below. We aren't doing anything but displaying the uploaded image in this walkthrough. In reality, there is much more to do with this URI, like user profile pictures, NFTs, subgraphs, and more.

Decentralized storage is restoring control to the owner of the data, and with the introduction to The Permaweb there is now a considerable amount of decentralized software we can create with Arweave and Bundlr. The more we build with these tools, the more control we have over creating the landscape we want and believe in. I hope you enjoy building with them as much as I now do.

Screen Shot 2022-05-22 at 9.33.30 AM.png

Arweave hosts all images used in this article - arweave.net/FtNpYIWTGD0JwM9NhQQCB5EjznZ7s4q.. arweave.net/8skBjH3_KSBtioHTWwe6F2pZ_CxSRTZ.. arweave.net/_XWV2FgRIt0GLHWo5Jge4mjQCSC6pFt.. arweave.net/Fcd6_KQFg1JhhhYRG-sqPTvy7RrGEWj..

Special thanks to

  • Nader Dabit for his tutorials on these topics, especially this one.
  • Arweave and Bundlr for making this as simple as it is.
  • Developer DAO for introducing me to the world of decentralization.
 
Share this