News

The Challenge of Consistency in Distributed Systems Achieving consistency across distributed systems is a notoriously difficult problem. The key reason is that, in a distributed environment, multiple nodes can independently make changes to the same piece of data. When different nodes hold different versions of this data, deciding how to reconcile these differences without losing valuable updates or introducing conflicts becomes a complex challenge. Traditional approaches often require coordination mechanisms, such as consensus algorithms (like Paxos or Raft), to ensure consistency.

In the 1960s psychologist Stanley Milgram conducted an influential experiment that revealed something amazing about human relationships. Milgram chose people at random in cities like Kansas and gave each a letter with the address of someone they didn’t know in Boson, Massachusetts. They were instructed to get the letter to that person but only by sending it to someone they know personally, who would send it to someone they know personally - and so on.

This Week’s Progress We focused on stabilizing network operations after recent updates. Most major issues have been resolved, but a key challenge remains with the WebSocket API: WebSocket Connection Stability: Issue: WebSocket connections occasionally drop, particularly during contract updates. This may be due to the lack of a keep-alive mechanism or another issue with how the client handles connections. Next Steps: Investigating whether periodic ping messages can prevent these disconnections. The application and node will also be tested to ensure they handle connections robustly.

Current Progress: Significant cleanup has been done, focusing on resolving issues with the transport layer and dependencies. Transport layer is working well, and the remaining issues are expected to be fixed within the next few days. The handshake handler has been thoroughly tested, with only minor remaining issues that are actively being addressed. A new monitoring and logging tool is almost ready and will be integrated soon. Next Steps:

Progress Overview: Improved Connectivity: Recent changes have allowed peers to establish more connections, even when multiple gateways are involved. Although not fully complete, connectivity between peers is progressing well, and extensive testing will continue to ensure robustness. UI Enhancements: A new UI is being developed to monitor and debug the network. This will aid in integration testing, making it easier to identify and fix issues in real-time, and will be helpful as we prepare for the release.

Key Achievements: Transport Layer Improvements: The network joining through a gateway works fine with most errors resolved. Nodes are acquiring connections, and the retry mechanism ensures successful connections even when initial attempts fail. Unit Test Success: Most transport layer unit tests are passing, with the system able to establish connections after retries. Random packet drop simulations highlight some intermittent failures, but overall functionality is stable. Connection Debugging: Logs show nodes progressively acquiring connections over time.

Just a brief update this week as we work towards the alpha release. Gateway Connection Handling After a few more fixes, connections with gateways are now handled smoothly. Although transport may still fail sporadically, we now appropriately retry connections, which resolves many of the previous issues. Regular Peer Connections Regular peers can now connect with each other! While there is still some weirdness that we are investigating, the connections are finally working as expected.

What’s Working: Peer-to-Peer Connectivity: Gateways and peers can now successfully connect and communicate with each other. While there are still minor issues, the main structure is operational, and communication between nodes works as expected. Message Transmission: Messages can be sent between peers, and errors that do occur generally resolve themselves as the system retries connections. Unit Tests: Existing unit tests for peer connections are passing, indicating stability in fundamental network components.

Key Progress: Major Refactor Completed: Refactor focused on initial connections between nodes via the gateway. Addressed numerous issues that were causing problems in the network. Integration testing has been improved, leading to faster feedback for changes. Integration Testing Improvements: Fixed various errors in integration code that were causing issues during testing. Adjustments have sped up the testing process significantly, allowing for quicker iterations. Current Blockers: Transport Layer Issues: There is a problem with the handshake process between the gateway and peers.

In this interview, Michael from FUTO sits down with Ian Clarke to discuss the revolutionary concept of Ghost Keys. They explore how these anonymous, verifiable identities could address some of the Internet’s foundational flaws and delve into the future of decentralized, Cypherpunk-inspired reputation systems. Also available on Rumble 𝕏

There Is No Negative Trust on the Internet On May 3rd, 1978, Gary Thuerk, a marketing manager at Digital Equipment Corporation, sent the first spam email to 400 people. It was an invitation to a product demonstration for the DEC-20 computer, and the reaction was immediate and negative. Nearly 50 years later, this same flaw in the internet’s design has given rise to more significant issues. Today, AI-driven bots not only overwhelm us with spam but also manipulate social and political discourse at scale.

This week we’ve been focussed on a crucial refactoring task to improve how we manage network connections. The goal is to make it easier to isolate bugs by separating the connection handling logic from the transport layer so they can be tested independently. What’s Changing: Decoupling Connection Handling: We’re separating the connection handling code from the transport layer. This change allows us to test connection states on their own, without involving the transport mechanisms.

Ian Clarke, the creator of Freenet, explains how we solve problems like efficiently finding data, adapt to changing network conditions, and managing a peer’s resource usage. Q&A includes how Freenet compares to other networks, the history of Freenet, and how Freenet adapts to geography. Also available on Rumble 𝕏

Freenet Chat Development: Ian has been working on the Freenet chat system and shared a specification document. He decided to focus on a web-based interface rather than a command-line interface due to ease of implementation. A significant topic was the method for updating contracts within the network. Ian proposed a replace contract field that allows for contract updates signed by the contract owner, similar to HTTP 301 redirects. Technical Challenges and Solutions:

Focus on Connection Stability and Transport Improvements Our primary focus has been on enhancing the stability and functionality of the connection and transport layers within Freenet. Ignacio has dedicated significant effort to address issues related to the connect operation and transport mechanisms. We’ve identified and resolved several bugs, ensuring that connections are maintained properly and cleaned up when lost. Although we haven’t fully tested all scenarios, the connect operation is now functioning as expected.

In our latest dev meeting, we dove into the recent updates and challenges with the Freenet network protocol. Ignacio shared the latest on our efforts to boost node connectivity and stability across the network. What’s New: Stability and Bug Fixes: We spent a good chunk of this week squashing bugs to make network connections more stable. Ignacio explained the technical hurdles we’re tackling to keep connections between nodes reliable. It’s tricky but we’re making headway.

Summary: In this week’s Freenet developer meeting, Ian Clarke and Ignacio Duart discussed significant advancements and remaining challenges before getting the network up and running. The primary focus was on refining the connection and configuration processes within Freenet’s system. Key highlights include: Configuration Management: The developers have implemented a system to set and save default configurations, which are crucial for initializing and maintaining stable operations after restarts. The configuration files are currently managed using TOML due to its robust support in Rust.

We had a detailed technical discussion focusing on various aspects of our project, Freenet. Here’s a breakdown of the key points: Contract Update Mechanism: We tackled the contract update mechanism, crucial for handling the state (the associated data) of contracts in our key-value store. This involves understanding how updates are initiated by applications, the merging of states, and the process of sending updates to subscribers. Update and Merge Process: We discussed the specifics of how updates work, particularly focusing on the ‘put’ operation.

Zero-knowledge proofs are one of the most significant developments in cryptography since the invention of public-key crypto in the 1970s. But what exactly are they, what are they useful for, and what is their relevance to Freenet? What Are Zero-Knowledge Proofs? In essence, zero-knowledge proofs (ZKPs) allow one party to prove that they have some secret information, but without revealing anything about the information itself. For example, you could prove that you have some data that satisfies a certain condition like hashing to a particular value, without revealing the data itself.

Ian Clarke explains how to build apps on Freenet, including the basic components of Freenet’s architecture like contracts, delegates, and user interfaces. Also available on Rumble 𝕏

Traditional approaches to rate-limiting the creation of things like coins, tokens, and identities in distributed networks rely on computational (proof-of-work) or financial barriers (proof-of-stake) as a source of scarcity. While effective in some contexts, these methods are wasteful and unfairly favor those with more resources. What is Proof-of-Trust? Proof-of-Trust offers an alternative by utilizing the scarcity of reciprocal trust between individuals as the primary resource for rate-limiting various network activities.

Ian sat down with Louis Rossmann to discuss the history and motivation for Freenet, and why he decided to re-architect it.

Every node in the Freenet network has a location, a floating-point value between 0.0 and 1.0 representing its position in the small-world network. These are arranged in a ring so positions 0.0 and 1.0 are the same. Each contract also has a location that is deterministically derived from the contract’s code and parameters through a hash function. The network’s goal is to ensure that nodes close together are much more likely to be connected than distant nodes, specifically, the probability of two nodes being connected should be proportional to 1/distance.