Open Source Filecoin Application Adventure

I want to make Filecoin storage deals easier. This goal has become a serious obsession because I think Filecoin is very fun when you can engage with the network easily. The Filecoin Network is insanely cool future technology so if you don't know much about it, click here to learn more. From my perspective there are three types of people caught in the tidal wave of wanting to make Filecoin storage deals, and it is completely independent of their context or the outcome they are trying to achieve.

  • 𓀩 ➝ You are not a technical person and you just want to make a Filecoin storage deal. Fire and forget.

  • 𓁅 ➝ You identify as a systems programmer and you want to make a Filecoin storage deal with tools that feel familiar.

  • 𓀨 ➝ You are an applications developer and you want to connect APIs to make a Filecoin storage deal.

If we make this process easier, we can get to the fun part which is preserving humanities most valuable data. So with this sketch of an understanding, I went on what I consider an “Open Source Filecoin Application Adventure”, where our team (https://arg.protocol.ai) lead by @whyrusleeping iterated on some code, repositories and published a few websites. As you read about the prototypes (and hopefully view source), I hope they serve as examples to help you get started on your own Filecoin adventure. 🙏 Let's go!





Prototype № 1 — https://file.app

https://github.com/application-research/miners A website with Filecoin miner performance, activity, and data.



I assume people prefer to start with network information that reduces the complexity of what is required to understand a new ecosystem. I think https://file.app helps with that. Traditionally, we all have our ways to get data. At work, I observed a pattern where people were distributing methods to get this category of information from personal notes. This is a bash snippet for finding the Filecoin market deals for any given miner: 

curl --compressed https://marketdeals.s3.amazonaws.com/StateMarketDeals.json | jq '.[] | .Proposal | select ( .Client == "f0399929" )' | jq . | less



A co-worker had shared this snippet from a savvy Protofire contributor and it is effective, but it is not ergonomic for everyone. I also found people wished to know the cost of Filecoin relative to other popular services such as Amazon S3 or Google Cloud Storage. But since we launched Filecoin in Q4 of 2020, it was difficult to have every calculation available to the public. We had meaningful estimations but our observers wanted the real market data faster than we could provide. So https://file.app was born in spirit of determining the mathematics and data presentation out in the open. It made sense to figure out this information out in public since we are learning new things every day, and I think being open has paid off. Plus, really smart people have already provided useful feedback (Thank you Marteen).

Merging miner indexes from FilRep and Textile helped our reporting stay consistent. I am eager to add a few more indexes that can be cross referenced so our reporting stays accurate. If you know of any, please let me know!





Prototype № 2 — https://storage.market

https://github.com/application-research/next-storage-deal The easiest way to make a Storage Deal with Textile's infrastructure.



I'm a huge fan of Textile's infrastructure. It's getting better day by day. It is the reason why I can implement a Filecoin Storage deal that only needs two parameters. Before I arrived at this prototype, I thought I had a breakthrough with the user experience with a far more complicated form (5 fields with a ton of terminology). However, Juan Benet changed my mind completely. He suggested to reduce it to the simplest form possible. So I came to three actions:



Simple in usage? Yes. Simple as a concept? Maybe not.

Problem A) A lot of people tell me they have no clue what a CID is. I think that is correct, and bearable. You would not be playing around with Filecoin Storage Deals if you didn't have some experience with IPFS, SHA-256 hashing, and other ecosystem topics before. But everyone's concerns are directionally correct, socializing CIDs is an education hurdle we need to resolve eventually. And if you don't want to learn what a CID is, just use Slate. Problem B) Why do you need Filecoin storage if you can reach your pinned data on IPFS already? Trying to teach people that Filecoin Storage is still valuable even though their data is already accessible is challenging. For example, you can view this URL and see your files. It is hard to educate people that there are no guarantees with the cloud, but there are guarantees with the Filecoin Network, and the 1500+ miners that are verifying storage data. Human challenges aside, at least I can make successful deals with this prototype!







After 2 days there were 6 successful deals out of the 9 deals I started. Each storage deal will remain on the Filecoin network available for retrieval deals for half a year. Optimistically, the only reason why the others failed was because I ran out of DataCap. Curious about DataCap? Check out this link from our friends at Infinite Scroll. This was a fun process because the Filecoin's mining community is really friendly and kind, here are the miners I worked with: TippyFlits ➝ https://storage.market/f08403 ➝  Oxford, UK Stuberman ➝ https://storage.market/f01278 ➝ Michigan, USA NeonixAF ➝ https://storage.market/f019551 ➝ Birmingham, UK Funk ➝ https://storage.market/f014768 ➝ Singapore Wijnand & Hidde ➝ https://storage.market/f01240 ➝ Amsterdam, Netherlands Brian Y ➝ https://storage.market/f023971 ➝ Florida Phi-rjan ➝ https://storage.market/f023467 ➝ Oslo, Norway Nelson ➝ https://storage.market/f022142 ➝ New York, USA

Working with the miner community means getting a first hand glimpse of the miner experience and the problems that come with it. I began to wonder how any miner could share out that they were accepting deals? So naturally I put a prototype together:

  • You bring a CID with real data.

  • Visit their miner profile.

  • Paste the CID (or type if you're a crazy person).

  • Click "Make Filecoin Storage Deal".

If you know of any other websites that make Filecoin storage deals easier than this, please tell me :-) There is a lot more we can improve with https://storage.market, I'm really looking forward to the Filecoin+Metamask production release because that means everyone can use their own Filecoin Wallet without having to create an account. Until then, I've applied to get more DataCap so I can help more deals go through.





Prototype № 3 — Next-Bucket

https://github.com/application-research/next-bucket An example template application for managing Textile buckets.

The template application provides a really straight forward experience:



  • Generate a LIBP2P key

  • Click use

  • Get a Filecoin address, and upload anything you want to IPFS.

  • Make a Filecoin storage deal and track the progress.

I was really inspired by all of the websites that host their NFTs using IPFS. NFT Storage in general is all the craze right now. I love the service that Protocol Labs released called https://nft.storage/ (powered by Pinata). It is awesome and combined with this blog post it is going to make people understand and correctly use all of the infrastructure that has been built. Super cool and massive kudos to everyone involved. To participate on the side, I wanted to whip up an example that demonstrated using Textile's Buckets to build their own OpenSea or similar platform. Thanks to Daniel's help with the Metamask integration, anyone who has built a Web2 service before can build a Web3 service as well. Another option for DiD was Magic, which OpenSea also uses. This prototype ended up as a great way to inspect the Filecoin storage deals that were being made on https://storage.market, so I put a hosted version of the next-bucket template behind the subdomain https://manage.storage.market. From there you can click on any of the buckets and see the contents of the deals through an IPFS gateway.

I've really grown to like this interface, kudos to the IPFS team!





Prototype № 4 — Estuary

https://github.com/application-research/estuary-frontend A Filecoin storage deal gateway and website front-end.

Always save the best for last. Estuary is an insanely cool and very effective way to make Filecoin storage deals against many miners at once. I'm really glad Why let me work on this one. In addition we aren't using any tools from any partners, this prototype is completely hand rolled.

Ready for the steps?



  • Upload a file.

  • Obtain the CID, but you don't need to keep it.

  • Click "Dashboard".



It is really that simple. Once you get to the Dashboard you can see all the things you would ever want to see, we track all 16+ states of where a Filecoin storage deal can end up during the storage deal transfer and sealing process. For each deal there is enough data that you can use to look up additional metadata, but we will probably bake that into our interface.



In a lot of ways, this prototype ended up far more powerful than what I demonstrated above with Storage Market. 💯



I enjoyed this. Working on high information density websites is challenging, and I am not very good at it ;-P I love that the storage deals can happen in parallel and I love that we're able to make small file Filecoin deals. We are still actively designing and prototyping this product together so stay tuned for updates. Data retrieval is just over the horizon. 🌇





Interested in helping out?

Do you want to collaborate with us at https://arg.protocol.ai? Our small team loves sharing ideas with anyone who loves open source and making software, if that sounds like you and you are excited about a new future wave of internet technologies, please jump into our open source portal.

➝ Talk soon 𓀡

Hey! My name is Jim and I love to build websites. You can learn more about me here.

To reply you need to sign in.