Substratum Dev Team Show & Tell — 21-June-19 (i33)

Apologies to the community for missing last week’s Show & Tell Summary

Show & Tell weekly sessions are an excellent insight into the challenges and triumphs facing the Substratum development team. Each week a set of slides is released and a group conference is held with dialogue and examples of work completed, tabled and planned on the horizon!

This week has a handful of exciting fixes, the most important being an extreme performance boost, showing very fast network speeds compared to previous code base!

Abram Cookson the CTO, begins the meeting in B.J’s absence. As always, the meeting beings by outlining the Strategic Goals of v1.0.0 Substratum Node:

Priority 1 — Make the Internet Free (as in Freedom) and Fair

Priority 2 — Earn Cryptocurrency. Change the World

Tactical Focus Items are outlined listed and briefly mentioned as Work-In-Progress

The ‘Show and Tell’ segment this week explores not one, but SIX achievements!

The team was noticeably in great spirits, and were all excited to explain various parts of the work completed for the week.

It was also very nice to hear from a few other team members this week, who had really shown amazing persistence and progress with their cards.

Outlined below are the major items to digest this week!

Tell — Proxy Client Timeout charges for 0-byte

This refers to when a failed request doesn’t make it past the proxy client to the server, the route is not charged to the user (basically the request drops on the floor and is not charged for sending bytes relating to the request itself).

Substratum Base Version Number

This allows the Node software to know if other Node users are using a different version, which can help compatibility as the network and software develops in future. This version number will be in the payload in CORES packets so software can identify this.

Node GUI IP address field shows HTML content when 503 error — Justin Hohman

The GUI has been tweaked to allow more usability. When Node is activated in the GUI, the setting prompts for information as before, but won’t stall if info needs to be changed. The “Save” button now is “Stop & Save”, which lets you change your configuration or neighbor descriptor without freezing the GUI.

When you change the State from Serving to Consuming and vice-versa, it saves your External IP details and neighbor descriptor.

A bug was also removed that leaked HTML error output text into the descriptor field

SHOW — Node performance ‘not good enough’ for v1 — Justin Hohman

Developer Justin Hohman showed a demo on-screen of a 7-node cluster hosted in Google Cloud, being created with TNT (Test Net Tools). Before using this test Sub Network, speed is normally 150Mb/s on their Wifi network.

On the test network demo’d live, the speed test shows 58 Mb/s, which is impressive considering the team were running the live Zoom meeting at same time.

Before, our community testers were only getting close to 10Mb on cloud instances!

To create this massive performance boost, the accountant process was running together with the other processes with node. By separating the accountant process into its own thread, this removed database calls, and can run on its own without blocking the rest of what is going on in the Node software

SHOW — 10th Node Error — Justin Hohman

Justin spun up another 10 node network, (which was cool as they have an Easter Egg that names each node after a Pokemon!), and showed a visual graph of all the nodes connected successfully to each other.

There was a major bug that was causing issues when a 10th node was trying to connect to a single descriptor.

Now the testing shows this fix has prevented issues with the node network scaling out from a single node descriptor!

SHOW — Support HD Wallet addresses — Steve Swing

In a previous Show & Tell, it was demonstrated that a wallet mnemonic seed can be created using Node (the 12 words that generate public wallet addresses using BIP32). Now the Node software knows how to generate the mnemonic phrase and use this to derive both Consuming and Earning wallets addresses, for use immediately within the Node.

The consuming wallet will need a private key, as this is used to sign the transactions when you consume on the network.

Using the command line, Steve showed this process, and also how to encrypt the wallet with an additional password.

Basically, earning and consuming wallets can be created, encrypted and stored within the Node from inside the software, making this process streamlined for users who want to start fresh with new wallets to hold and spend their SUB tokens from running Node!

Its quite amazing how the trajectory is steady, with target of only 2 weeks for completion of Node v1.0.0 Release Candidate 1!

Abram spoke so proudly of the team and how they went full steam ahead, pumping out the completed work this week like a well oiled machine.

“6.99 cards completed” — Abram Cookson

6 cards were completed (almost seven!) this week, with 3 being bug cards and 3 planned cards.

Solid utilization with the team being equally productive on both bug cards and planned work/feature cards.

Overall it was an amazing week! Not only did the team achieve some important fixes that were hindering the Node network expansion, but also a critical performance fix that now helps users see great speeds across the network in testing. Other comments from them hinted at even further performance tweaks that will see even more boosts.

Stay tuned for another Show & Tell next week with the Substratum Developers!

For those who want more technical insight, I tried to capture most of the Guest Open Space chatter that was really interesting. If you don’t want to ‘geek out’, you can stop reading here!

Guest Questions/Comments during meeting (paraphrased!):

Cuba had an emergency BBQ — he gently reminded us which is rich baritone voice that he only gets to enjoy 3 days of sun a year where he lives!

Guest: Is next week the ‘payout week’ for node?

Steve: Payout card is being worked on now

We have the current ability to detect payouts, but not blockchain confirmations right now.

Hope that it will be completed quickly.

Crypt0_Kiwi: Is 10th node bug truly fixed? In Brian’s honour can we see a demo of 10 nodes?

Justin spun up 25 nodes immediately

Showed the visual digraph of the nodes — all nodes were connected!

Further testing needed with Node dropping old and rejoining etc. As opposed to all nodes being started at same time.

Cuba: big shout out to the testing group for all their work!

Team: Agree! Internal Testing group praised!

Dan Wiebe: Preach brother!

Abram Cookson: I feel like the ‘feedback loop’ is the best out of any company I have been part of.

Maybe because of the open source nature, but it’s quite awesome, unlike how it can be in a corporate environment or conference

T: How are you dealing with wallet addresses?

Only ones created within node can be used?

Steve: for now we are just dealing with mnemonic seed.

We have a RC2 card — so you can recover a wallet

We have designed node in order of what a user would use with running the software.

ETH addresses are compatible with Ropsten and mainnet, so we assumed users wouldn’t be using their real SUB and SUB wallets as the version roll-out.

In the next few updates, users should be able to import wallets with a json key store.

It may also be possible for users using MEW type wallets and not hardware wallets.

We will have a way to do this with CLI or a config setting in the .toml file, and use password to decrypt it

Another input to be developed would be to recover mnemonic phrase (same as the one on HD wallet), to initialize the wallet and derive the wallet within node using the passphrase (13th word)

Crypt0_Kiwi: If a large HTML download fails, will the partial request have a zero charge?

Dan: we are charging by CORES package. So large files are not packed into one cores packet

If a large file of 10gb aborted after 2gb for example, the CORES packets were routed and the other nodes routing have no idea that things failed.

So they are charging for routing.

At moment, no — if someone routes or transmits to you, you get charged for it.

Steve: bytes are charged as transferred, if browser doesn’t save it then that’s that. ISP is the same with data caps. If your computer drops the data after it has been recieved, ISP still counts it.

Question from Microoo on Telegram:

What happens when originate node makes a request of downloading a huge file and then cancels or disconnects? Do hoppers and exit node get notified of the cancellation?

Dan — working on that card now with Kevin.

If one end of connection gets closed, the originating node will translate that closure into a CORES package and transmit it to the exit node, informing it to close this stream on the transmitting end. So it will interrupt the data stream.

So the books should be kept in sync in terms of payments for routing and requests sent.

Kevin brought up another large node cluster and showed it on visual digraph

Same test but did daisy chain of connecting nodes (one connected to each one in sequence as they join)

Bridge bearer structure looks good. No errors could be seen.

Previous demo in the Show was all the nodes connecting to the same single node on startup.

Marcel: Wanted brief explanation of the problem with the 10th node failure

Kevin: Issue was that once you got to 10 nodes, the new guy was being introduced kind-of in a chain.

It would debut to 1st node who says “Well I’m full, but here is this other guy.”

The other guy would connect and say well you need another neighbour, then in a chain this kept continuing.

But the Node was connected, but did not inform the others that they were connected. So introduction gossip kept happening

When passes were happening, that situation wasn’t being detected.

Crypt0_Kiwi: It seems that nodes with 5 neighbors are rare to see in the new visuals

Abram: It’s like a rare pokemon! X-D

Get your masterball out!

Disclaimer: I am not an official Substratum team member — I am a community moderator. This write-up is interpretation of the Show & Tell meeting and data released from the team. None of this is to be taken or construed as investment-related or as financial advice.

Substratum Dev Team Show & Tell —7-Jun-19

Show & Tell weekly sessions are an excellent insight into the challenges and triumphs facing the Substratum development team. Each week a set of slides is released and a group conference is held with dialogue and examples of work completed, tabled and planned on the horizon!

This week revealed completed work on an exciting critical component for monetization — the integration of wallets securely into Node (with mnemonic phrases)

B.J. Allmon lead developer begins the meeting, as always, outlining the Strategic Goals of v1.0.0 Substratum Node:

Priority 1 — Make the Internet Free (as in Freedom) and Fair

Priority 2 — Earn Cryptocurrency. Change the World

Tactical Focus Items are outlined listed and briefly mentioned as Work-In-Progress

B.J. mentioned specifically that work has just started on node encryption in the last week or so, which is extremely important for a secure testnet and ultimately the first Release Candidate (RC1).

Also clarified at this point, was that before a ‘Version 1’ (v1) is finally released, team and community have to run the Release Candidates on testnets and identify things that are not on the planned scope, including defects, bugs and hardening.

Storymapping for Substratum Host has just begun, which is reliant on Node to even be possible.

Normally the ‘Show and Tell’ segment takes up the bulk of each weekly meeting, but this is the sparsest slide we have seen since these meetings have been broadcast!

The mnemonic phrase item actually took several weeks to finally finish — which is basically integrating how wallets will be connected into Node itself, both for earning and consuming by users.

But this goes to show how difficult the items are right now that need to be completed.

Outlined below are the two major items to digest this week!

Stop sending useless intros via Gossip — Dan Wiebe:

This card was identified by testers last week, and was moved forward because the issue itself could create further problems in future node-network building. When Node itself sends intro or passing Gossip (basically the function that says — “Hey Node B — contact this new Node C!”), the code has not been checking whether the new node was already known to the others.

This completed card now provisions that there should never be a node that has less than 2 neighbors (unless there is a true network problem)

This wasn’t prioritized for RC1, but since the testing community saw the issues this was creating, the team prioritized this to fix this.

SHOW — Generate BIP39 master seed demo — Steve Swing:

Steve demonstrated on screen the new functions integrated with blockchain wallets.

This can be considered one of the most critical and exciting components as this is the true core of monetization! — without it there is no way to earn and pay SUB tokens!

This card dealt with coding a number of command line (CLI) parameters, and allowing users to integrate wallets into the Node software.

The component is designed to allow both Hardware (think Ledger or Trezor) and Software based wallets (think Metamask or online MyEtherWallet) to be used while running a Node.

The code itself allows a BIP39 mnemonic phrase to be input, or to even generate a new mnemonic phrase from within the code (which is used to produce unique public wallet addresses).

The Testnet (Ropsten on ETH) will default to a 12 word-count for the mnemonic phrases, so it compatible with wallets like Metamask.

Mainnet will default to 24 word phrases.

“We knew early it would be a tough week.” — BJ

For the first time in 10 weeks, the forecasted weeks counter did not reduce.

It remains at 4 weeks.

The remaining cards are very tough and are complex — but the challenge also relates to the few number of cards left to work with.

Again this is a moving target!

All work was on the 2 ‘Planned’ cards this week, hence the 100% utilization in green.

There has been talk around the community about why the team seems focused on building all the components on the command line.

B.J. reinforced that these features are the underpinning functionality, so the GUI can even be built. So it has to be strategic, and have core features in place for Node on the system code side.

The team still very much cares about user friendliness and GUI development!

Now, the UI features can be built up including creating and selecting an earning/consuming wallet, user settings and more.

Overall it was an interesting week. B.J. did sound a little frustrated about week countdown remaining the same, but simultaneously proud that this has not happened in 10 weeks with the development! The critical component of the wallet integration, and parallel development with the blockchain services and referencing, really shows that monetization is being fleshed out successfully. It sounds like in the coming week a minor release could be expected, along with more work on encryption and blockchain integration.

Stay tuned for another week with the Substratum Developers!

For those who want more technical insight, I tried to capture some of the Guest Open Space chatter that was really interesting. If you don’t want to ‘geek out’, you can stop reading here!

If you don’t want to ‘geek out’, you can stop reading here!

Guest Questions/Comments during meeting (paraphrased!):

Steve: We have cards so that the GUI has the interface to prompt and recover the mnemonic phrases and generate them.

If you feel comfortable with your current wallet, you can import your recovery phrase you initialized your HD wallet with, into the node software and keep your SUB where it is to use in Node.

We (Node or dev team) don’t save your mnemonic phrase — We encrypt the resulting mnemonic seed produced by the phrase and your mnemonic passphrase (25th word if you have this)

Right now its fake encryption, so don’t use a real wallet yet if you are testing.

We will have an area where your initialized wallet is displayed in the future GUI

Brian: Couple things — great job on the critical fixes!

Have you upgraded to rust 1.35?

Steve: I think we are upgraded for the current builds

Brian: Are Port specifications in the UI?

The use-case is quite important, as users don’t want to open all the range of ports and use DMZ.

If I launch the UI, and it picks a port outside the range, it may not be able to do a full handshake into the network.

Port needs to be opened before initializing node

(so it’s almost a chicken vs egg dilemma)

Dan: Router should be able to be fully closed, and the entire handshake should work after you open the required port. If that is not the case, then we need to test for that.

Steve: if you are on a new machine that’s never used Node before, you can provide the clandestine port parameter in CLI, configure the script to use open port

Dan: it’s not in the UI now. But what might help you if you have used Node on CLI, it will have chosen a random port and stored it in the .db file. It will still use that port in future initializing every time Node is run. (unless you delete the database)

Brian: the challenge is we are rebuilding things frequently and having to wipe the .db file frequently when testing

BJ: no card yet for UI port assignment, but we will create one

Justin (developer): HTTPS traffic was not flowing through node. Routing was becoming sporadic. It was pulling different exit nodes — right now some HTTPS won’t work for TLS sessions (handshake). Team is fixing that, but it is a major challenge right now

BJ: Thing developers have noticed the most with current coding — ‘the fix is easy’ but ‘testing the fix’ is extremely difficult.

BJ made a NEW WORD! ‘Craftspersonship’ ?

Brian: Relating to HTTPS/TLS issues, thank you to Dan for writing up an explanation on how this should be working.

The current panics in testing community now are placeholder as some features aren’t implemented yet.

BJ: Panics are a problem, but we need to fix it. One is the blockchain url parameter not being specified.

Dan: Some issues still remain when blockchain url parameter isn’t provided. Even if you are serving-only, Node still will have to access blockchain services to verify blockchain info on your behalf. Blockchain info may be censored in your area, so the node network is leveraged to provide block info for payment verification etc.

NOTE: this is actually an aspect where users can earn more if their node is needed to fetch blockchain service data for other nodes who cannot access it, haven’t set it up in Node, or are restricted from those services in their country!

This is an aspect where your node may need to consume even though it is set to serving-only the network.

Steve: we need to explain — if you are serving-only and don’t have access to the blockchain data, you are trusting that other nodes have SUB to pay the services you are providing — if you can’t access blockchain, then you cannot verify if people can pay!

We don’t want that problem to exist! So we are writing the app so Node will have access one way or another.

If you don’t supply the URL it will check in time intervals due to blockchain API — we will design so it won’t crash/panic the node.

Andreas: A small request regarding public IPs — can Node recognize a node that may have a new or updated public IP, after it has already connected to the network?

Dan: Technically, the way Node is written now, it may not be necessary for Public IP to be a parameter. But we haven’t gotten around to addressing the fact that dynamic IP addresses will change and Node won’t adapt to this at this time.

Andreas: One idea to make it simpler — one solution some software have is to provide a DNS Entry instead of a static IP, so if we can just provide a DNS name for us, it can maybe be updated. But would need to write the code to update the database on the network

Steve: remember that the SQL database is not storing any data, just who and what you owe.

Dan: it also stores the port tables of delinquency, who owes what, and only stores wallet address. Not private keys.

Now stored in memory — there is hashmap of neighbourhood directory, IP addresses, neighbour lists, earning wallet addresses and a bunch of stuff about nodes, also that’s where we make the routes from.

When node goes down, that all goes away. Its ephemeral.

BJ: it was very intentional. All data we handle, we want to handle it with care. We want it to go away if a computer is stolen etc. What data we store is more ‘safe’ for storage from a safety perspective.

BJ: one thing tester noticed also, is that there is a performance boost by log filtering, and using log_level info instead of debug.

Disclaimer: I am not an official Substratum team member, just a community supporter. This write-up is interpretation of the Show & Tell meeting and data released from the team. None of this is to be taken or construed as investment-related or as financial advice.

Substratum Dev Team Show & Tell — 31-May-19

Show & Tell weekly sessions are an excellent insight into the challenges and triumphs facing the Substratum development team. Each week a set of slides is released and a group conference is held with dialogue and examples of work completed, tabled and planned on the horizon!

This week showed some exciting troubleshooting, and some major steps forward in both the monetization and neighborhood dynamics areas of Substratum Node.

With an air of battle hardened precision, B.J. Allmon lead developer begins the meeting by always outlining the Strategic Goals of v1.0.0 Substratum Node:

Priority 1 — Make the Internet Free (as in Freedom) and Fair

Priority 2 — Earn Cryptocurrency. Change the World

Tactical Focus Items are also listed and briefly mentioned as Work-In-Progress

Notably mentioned this week was how the team worked with Brian M, an amazing community node-running tester, who live tested a 5-node cluster with the developer team to troubleshoot. This led to 3 defect cards (the team uses Agile lean project management techniques) outlining fixes.

The team created an “Epic” story for RC1 critical fixes to correct some further routing bugs. These fixes are being implemented in next week or so.

Another minor version roll out is imminent (0.4.7)

RC1 is starting to be packaged and continuous work around this each and every day.

It was most exciting to hear comments about horizon planning for both Substratum Host and Cryptopay!

The actual Show and Tell segment takes up the bulk of each weekly meeting. Team Members take turns to ‘Tell’ the team and community of code commits or fixes, and also ‘Show’ on-screen changes or new implemented features to demonstrate their use.

Outlined below are the major points to digest from this week!

Creating a Sub Config File — Dan Wiebe:

Dev team implemented a ‘config file’ into Node software, because at the moment, the CLI (command line) startup command has a lot of parameters that need to be typed each time a user wants to run Substratum Node. (or integrate these parameters or settings into the environment)

Issue is, there could be prying eyes viewing a user’s command-line if typed each time. Not to mention it is cumbersome for testing and for users to simply start node on machines where their settings are not changing much.

Accountant scans & add delinquencies to ban — Dan Wiebe:

The Accountant module within SubNode keeps track of ‘who owes what, to who’ for routing, and from time to time refers to a receivables table of these values to ‘balance the books.’

Bills accumulate towards a threshold (based on a curve) where the bill is to be paid for consuming.

There is another threshold noted as a delinquency limit, and when these limits are exceeded, the accountant will add that consuming wallet to a ‘ban-list’ in memory, because their debt for consuming hasn’t been paid.

Based on this, a user using a wallet on the ban-list will not have their packets routed — meaning that their node will not be able to consume, unless they pay their dues, or consume across Node with another wallet with a positive balance.

Another upcoming change on this item will be that nodes with a consuming wallet on the ban-list will be removed from the ban-list when they have paid enough of what they owe.

Node Gossip not properly introducing 6th Node — Dan Wiebe:

Currently the node neighborhood is designed to have up to 5 immediate neighbors connected to each individual node.

A bug was detected through testing with Brian, where a 6th node was not being properly passed on to another nearby neighbor to attach to, when it tried to connect to a node that was ‘full’ (already had 5 connections). There was a discrepancy with passing on the newcomer node to another one nearby. The gossip protocol was not checking with the neighbors being passed to about whether the newcomer was already a connected neighbor to them. Due to this, new nodes joining were occasionally being passed back and forth and would not connect properly to neighbors nearby.

No more bootstrap nodes — Matt:

The bootstrap function has been deprecated and the code cleaned up to remove this. There were a few parameters and command references now removed.

Now nodes can connect freely with neighbors, by debut and passing gossip.

Removal of EntryDNS crashpoints — Dan:

During testing, there were Node crashes when an EntryDNS query failed trying to receive an invalid DNS request from a browser process. There was a fix put in so the Node doesn’t simply crash when this behavior occurs, it just gives an internal error to the EntryDNS module.

SHOW — Demo from BJ:

B.J. put the updated software GUI on-screen to show the changes. The updated SubNode GUI shows a much more streamlined interface. The forms only show the required fields to Serve and Consume, and there will be a ‘Config’ option in the Node menu to allow configuration of settings into the config file that was integrated.

Changing the Node status (between Off>Serving>Consuming etc) will retain the configuration that was inputted by the user.

Note — configuration settings only persist while the Node instance is running. It will be embedded in future versions.

Also in future version coming soon, there will be data shown in the UI like statistics of routing, earnings etc

Account Payment Detection — Matt:

This module scans the blockchain for the highest block number, and detect recent transactions to update the accountant module and payment statuses (keeping the books up to date!). Block number stored in database, so scanning can be done in forward fashion.

Over last 9 weeks, the team has seen constant progression, with each weekly iteration drawing a week closer on the countdown.

The team ‘just scraped through’ this week on the countdown, with the tech debt involved this week working on the 3 defect cards (it’s Brian’s fault lol).

“Team worked really hard, and getting stuff pushed through.” — BJ

This is a moving target — scope can change, people could be off sick etc!

“This week reflects the work we did with the testing community… the 29% and defect fixes…”

BJ explained that the team is now ‘playing whack-a-mole’ on remaining issues we don’t want in version 1.

57% capacity was on the ‘planned work’ side so that looks really good.

Guest Questions/Comments during meeting:

From Brian — the team ‘kicked butt’ with the bugs! Things looking good with 5-node and 6-node clusters in current tests! Routing works perfectly with HTTP & HTTPS!

Will continue to test in isolated test environment.

Had to delete the sqlite database (node-data.db) when restarting nodes and building new networks.

Steve Swing — Name and location of database (.db) file has stabilized so it may be prevalent, but is easy to work around now with that.

Overall seems like another weekly ‘notch on the belt’ for the project timelines, with a successful work-week accomplishing fixes and cards leading to v1 RC1 (Release Candidate 1). The sense of relief in the discussion was noticeable from the team, since it was a short work week in the USA, but the meeting was also quieter than usual as some regular attendees were not present.

Stay Tuned for the next update!

Disclaimer: I am not an official Substratum team member, just a community supporter. This write-up is interpretation of the Show & Tell meeting and data released from the team. None of this is to be taken or construed as investment-related or as financial advice.