Windows 10 Node Teaser

Hey Substratum Community – This is a late night teaser of the latest build from source. It is not backwards compatible with pre-built 0.4.8 binaries. The left screen is an ssh session to one of my AWS linux nodes. The right screen is a Windows 10 VM running on my home desktop. When traffic is generated on that node, the log becomes very chatty. This video will demonstrate the latest core functionality. It will show the decryption of the earning wallet as well as the password prompt for the consuming wallet feature.

Final payment services cards are in the works. We are anticipating next release to include the RC-1 tag, which will require testers to have their Ropsten $HOT tokens available for testing all blockchain payment features.

Last, my Windows 10 VM is running in a high trace/debug level. The pop up screen at the end is expected.

Substratum Dev Team Show & Tell — 19-July-19 (i37)

Show & Tell weekly sessions are an excellent insight into the challenges and triumphs facing the Substratum development team. For those following these Show & Tell discussions, the format of the progress breakdown has just gotten slightly more tense and exciting, because the team is down to the last handful of critical items (cards) to move to the first Release Candidate (RC1) of the SubstratumNode

As always, B.J. Allmon heads the meeting by outlining the Substratum Node Version 1 Strategic Goals:

  • Priority 1. Make the Internet Free (as in freedom) and Fair. — “As a SubstratumNode user I can allocate my spare computing resources so that the Internet can be a free and fair place for the entire world.”
  • Priority 2. Earn Cryptocurrency. Change the world. — “As a SubstratumNode user I earn $SUB as content is routed through my node so that I am incentivized to make the Internet free and fair.”

Many of the Tactical Focus Items listed have been done for a while. Most of these have been done for the purpose of RC1 and the others that are Work-In-Progress will be built-out further after RC1 to provide more in-depth features.

GUI (Graphical User Interface) updates have notably been challenging and involve a lot of work.

The ‘Show and Tell’ segment

The Development team review the completed work for the week, including Telling the story of each card, or Showing the completed feature.

This week was interesting to hear all the inside dynamics about the wallet generation from Dan Wiebe and accessing it from Node itself. Most engaging was the number of on-screen demos from Command-Line

Correct handling of wallet password from gui

This was an item that Dan noticed. The wallet password is used in multiple places in Node, but it was not correctly being referenced in all places in the software.
Once the wallet details are unlocked with the user password, now it is properly referenced in all areas.

use default fixed service rate pack

Right now there isn’t a configurable rate (cost per CORES packet routed) yet integrated into Node. This card enforces if someone happened to modify the source code to charge some arbitrary rate per CORES packet, Node then reverts to the default rate (which will be specified later on by the user in future Node versions).

provide better clap cli help – clandestine port and neighbors

Help file onscreen in command-line has been built out, with more info and verbiage for users.

Dan also showed this in his Show session, illustrating that the Help screen update provides more info for users when using the wallet integration commands.

show – windows subnode cli doesn’t print out anything – dan wiebe

Dan Wiebe took the lead on the SHOWs this week and first showed a demo of the use of Substratum Node from CMD command-line

Dan shared on-screen after some deliberation (BJ casually walked over, as a battle-scarred Zoom meeting veteran, and fixed it for him)

This is great, as a lot of users and testers are Windows based!

show – consuming private key into database + blockchain: pay bills!- dan wiebe

This segment was both lengthy and very thorough. This area of cryptography is the foundation of blockchain in both Bitcoin, Ethereum and many other chains. As much of the wallet address and key pair dynamics are very elaborate, you might find this other post useful that I am about to publish (link to be added) – its quite a lot to wrap one’s head around!

In a nutshell, wallet integration is coded within Node software, so that it can generate mnemonic seeds and allows the BIP32 cryptography to generate key pairs for wallets on the ETH blockchain.
While you can both generate new wallets and recover your own using your mnemonic seed and passphrase (and use wallet private key), this entire integration means that things can be simplified for users once the UI is built out. It also allows users to use wallets from many different origins including Hardware wallets like Trezor and Ledger.
This is critical for adoption by both new entrants to the crypto sphere, but also to others who are not familiar with managing wallets, and use the assistance of user-friendly platforms and hardware.

While the demo showed the many ins and outs of the way Substratum Node interacts with wallet generation, recovery and integration, Dan also demonstrated what is shown and stored in the database file. (stored in sqlite3 format – node-data.db)
With both security and monetization factors in mind, different parts of wallets and keys are stored in the node-data.db database file itself, depending on what is specified.

It does not show the consuming wallet private key (as that could be used to drain the wallet of funds by a malicious party!)

By providing the consuming private key to SubNode when it runs, it does not display the mnemonic seed or the consuming wallet derivation path, unless this is specified by the user purposefully. If the seed is displayed, it is displayed in encrypted format, using the user-defined password.

That said, the consuming wallet public key and earning wallet addresses can also be stored in the database. This is integral in the monetization model, because a node will have to prove to others that are involved in data routing and transactions, that it:

  • Has a correct private key that matches the public key for the Consuming Wallet
  • Has an earning wallet address stored (so others can ‘pay’ it for routing)

This was also confirmed on-screen showing the Ian Coleman BIP32 Mnemonic Generator, which confirms that the mnemonic BIP32 generator and functions all work correctly.

Also critical in this area, it was demonstrated that YOU CANNOT CHANGE THE EARNING WALLET once your Node is initialized. The main reason for this is because it will interfere with the accounting module and the bookkeeping.
If you do intend on changing the earning wallet, you need to delete the database file and specify all wallet details again. In this case you will lose all account receivables that are owed to you for routing.

Additionally, the public and private keys are matched during the initialization of Node, so if the user does change the private key when starting up, Node will detect that it is different to the public key that is stored (as the private key is not stored in the database for security reasons outlined above).

Lastly, it was shown that language support is provided for wallet generation and command line for Chinese, Japanese, French, Italian, English and several others.

As explained several weeks ago, the tracking and completion projections have changed as the final few cards are in motion. Each card is critical to be completed for RC1, and 2 cards are blocked by one in progress now.

The reason the Todo card count is at 2.5 is there is a card that was finished which was going through review on Friday, right after the Show & Tell session.

Happy friday! Here is today’s show and tell deck. Quick heads up. The “Blockchain: pay bills” card is in team review right now. It will be made public once review is completed and any minor tweaks are made if necessary.

B.J Allmon

With 3 cards completed during the week, this team is not stopping the momentum – the tracking method has changed for the public to view the progress based on cards remaining.

To keep all the team utilized, there are some RC2 cards that are being worked on while other coding pairs (pairs of developers working together on card items) are busy with the RC1 cards in progress.

Team Capacity Utilization overall is interesting right now because RC1 is imminent – the yellow area shows some parts of the team are working on background RC2 items, to ensure that all team members are productive on the initiative.

The bugs that were worked on took up the 33% capacity were relating to the wallet monetization components and the Windows usability of the software

Things are getting closer and closer as RC1 work is checked off and implemented!
Stay tuned for another update next week!

If you have enjoyed all the Show & Tell summaries thus far, please donate to the cause below, so the Substratum Wiki community can continue to curate, host and push out this great information for everyone!

Disclaimer: I am not an official Substratum team member — I am a community moderator. This write-up is interpretation of the Show & Tell meeting and data released from the team. None of this is to be taken or construed as investment-related or as financial advice.

ETH donations much appreciated, and will go directly to support hosting the website:


Substratum Dev Team Show & Tell —7-Jun-19

Show & Tell weekly sessions are an excellent insight into the challenges and triumphs facing the Substratum development team. Each week a set of slides is released and a group conference is held with dialogue and examples of work completed, tabled and planned on the horizon!

This week revealed completed work on an exciting critical component for monetization — the integration of wallets securely into Node (with mnemonic phrases)

B.J. Allmon lead developer begins the meeting, as always, outlining the Strategic Goals of v1.0.0 Substratum Node:

Priority 1 — Make the Internet Free (as in Freedom) and Fair

Priority 2 — Earn Cryptocurrency. Change the World

Tactical Focus Items are outlined listed and briefly mentioned as Work-In-Progress

B.J. mentioned specifically that work has just started on node encryption in the last week or so, which is extremely important for a secure testnet and ultimately the first Release Candidate (RC1).

Also clarified at this point, was that before a ‘Version 1’ (v1) is finally released, team and community have to run the Release Candidates on testnets and identify things that are not on the planned scope, including defects, bugs and hardening.

Storymapping for Substratum Host has just begun, which is reliant on Node to even be possible.

Normally the ‘Show and Tell’ segment takes up the bulk of each weekly meeting, but this is the sparsest slide we have seen since these meetings have been broadcast!

The mnemonic phrase item actually took several weeks to finally finish — which is basically integrating how wallets will be connected into Node itself, both for earning and consuming by users.

But this goes to show how difficult the items are right now that need to be completed.

Outlined below are the two major items to digest this week!

Stop sending useless intros via Gossip — Dan Wiebe:

This card was identified by testers last week, and was moved forward because the issue itself could create further problems in future node-network building. When Node itself sends intro or passing Gossip (basically the function that says — “Hey Node B — contact this new Node C!”), the code has not been checking whether the new node was already known to the others.

This completed card now provisions that there should never be a node that has less than 2 neighbors (unless there is a true network problem)

This wasn’t prioritized for RC1, but since the testing community saw the issues this was creating, the team prioritized this to fix this.

SHOW — Generate BIP39 master seed demo — Steve Swing:

Steve demonstrated on screen the new functions integrated with blockchain wallets.

This can be considered one of the most critical and exciting components as this is the true core of monetization! — without it there is no way to earn and pay SUB tokens!

This card dealt with coding a number of command line (CLI) parameters, and allowing users to integrate wallets into the Node software.

The component is designed to allow both Hardware (think Ledger or Trezor) and Software based wallets (think Metamask or online MyEtherWallet) to be used while running a Node.

The code itself allows a BIP39 mnemonic phrase to be input, or to even generate a new mnemonic phrase from within the code (which is used to produce unique public wallet addresses).

The Testnet (Ropsten on ETH) will default to a 12 word-count for the mnemonic phrases, so it compatible with wallets like Metamask.

Mainnet will default to 24 word phrases.

“We knew early it would be a tough week.” — BJ

For the first time in 10 weeks, the forecasted weeks counter did not reduce.

It remains at 4 weeks.

The remaining cards are very tough and are complex — but the challenge also relates to the few number of cards left to work with.

Again this is a moving target!

All work was on the 2 ‘Planned’ cards this week, hence the 100% utilization in green.

There has been talk around the community about why the team seems focused on building all the components on the command line.

B.J. reinforced that these features are the underpinning functionality, so the GUI can even be built. So it has to be strategic, and have core features in place for Node on the system code side.

The team still very much cares about user friendliness and GUI development!

Now, the UI features can be built up including creating and selecting an earning/consuming wallet, user settings and more.

Overall it was an interesting week. B.J. did sound a little frustrated about week countdown remaining the same, but simultaneously proud that this has not happened in 10 weeks with the development! The critical component of the wallet integration, and parallel development with the blockchain services and referencing, really shows that monetization is being fleshed out successfully. It sounds like in the coming week a minor release could be expected, along with more work on encryption and blockchain integration.

Stay tuned for another week with the Substratum Developers!

For those who want more technical insight, I tried to capture some of the Guest Open Space chatter that was really interesting. If you don’t want to ‘geek out’, you can stop reading here!

If you don’t want to ‘geek out’, you can stop reading here!

Guest Questions/Comments during meeting (paraphrased!):

Steve: We have cards so that the GUI has the interface to prompt and recover the mnemonic phrases and generate them.

If you feel comfortable with your current wallet, you can import your recovery phrase you initialized your HD wallet with, into the node software and keep your SUB where it is to use in Node.

We (Node or dev team) don’t save your mnemonic phrase — We encrypt the resulting mnemonic seed produced by the phrase and your mnemonic passphrase (25th word if you have this)

Right now its fake encryption, so don’t use a real wallet yet if you are testing.

We will have an area where your initialized wallet is displayed in the future GUI

Brian: Couple things — great job on the critical fixes!

Have you upgraded to rust 1.35?

Steve: I think we are upgraded for the current builds

Brian: Are Port specifications in the UI?

The use-case is quite important, as users don’t want to open all the range of ports and use DMZ.

If I launch the UI, and it picks a port outside the range, it may not be able to do a full handshake into the network.

Port needs to be opened before initializing node

(so it’s almost a chicken vs egg dilemma)

Dan: Router should be able to be fully closed, and the entire handshake should work after you open the required port. If that is not the case, then we need to test for that.

Steve: if you are on a new machine that’s never used Node before, you can provide the clandestine port parameter in CLI, configure the script to use open port

Dan: it’s not in the UI now. But what might help you if you have used Node on CLI, it will have chosen a random port and stored it in the .db file. It will still use that port in future initializing every time Node is run. (unless you delete the database)

Brian: the challenge is we are rebuilding things frequently and having to wipe the .db file frequently when testing

BJ: no card yet for UI port assignment, but we will create one

Justin (developer): HTTPS traffic was not flowing through node. Routing was becoming sporadic. It was pulling different exit nodes — right now some HTTPS won’t work for TLS sessions (handshake). Team is fixing that, but it is a major challenge right now

BJ: Thing developers have noticed the most with current coding — ‘the fix is easy’ but ‘testing the fix’ is extremely difficult.

BJ made a NEW WORD! ‘Craftspersonship’ ?

Brian: Relating to HTTPS/TLS issues, thank you to Dan for writing up an explanation on how this should be working.

The current panics in testing community now are placeholder as some features aren’t implemented yet.

BJ: Panics are a problem, but we need to fix it. One is the blockchain url parameter not being specified.

Dan: Some issues still remain when blockchain url parameter isn’t provided. Even if you are serving-only, Node still will have to access blockchain services to verify blockchain info on your behalf. Blockchain info may be censored in your area, so the node network is leveraged to provide block info for payment verification etc.

NOTE: this is actually an aspect where users can earn more if their node is needed to fetch blockchain service data for other nodes who cannot access it, haven’t set it up in Node, or are restricted from those services in their country!

This is an aspect where your node may need to consume even though it is set to serving-only the network.

Steve: we need to explain — if you are serving-only and don’t have access to the blockchain data, you are trusting that other nodes have SUB to pay the services you are providing — if you can’t access blockchain, then you cannot verify if people can pay!

We don’t want that problem to exist! So we are writing the app so Node will have access one way or another.

If you don’t supply the URL it will check in time intervals due to blockchain API — we will design so it won’t crash/panic the node.

Andreas: A small request regarding public IPs — can Node recognize a node that may have a new or updated public IP, after it has already connected to the network?

Dan: Technically, the way Node is written now, it may not be necessary for Public IP to be a parameter. But we haven’t gotten around to addressing the fact that dynamic IP addresses will change and Node won’t adapt to this at this time.

Andreas: One idea to make it simpler — one solution some software have is to provide a DNS Entry instead of a static IP, so if we can just provide a DNS name for us, it can maybe be updated. But would need to write the code to update the database on the network

Steve: remember that the SQL database is not storing any data, just who and what you owe.

Dan: it also stores the port tables of delinquency, who owes what, and only stores wallet address. Not private keys.

Now stored in memory — there is hashmap of neighbourhood directory, IP addresses, neighbour lists, earning wallet addresses and a bunch of stuff about nodes, also that’s where we make the routes from.

When node goes down, that all goes away. Its ephemeral.

BJ: it was very intentional. All data we handle, we want to handle it with care. We want it to go away if a computer is stolen etc. What data we store is more ‘safe’ for storage from a safety perspective.

BJ: one thing tester noticed also, is that there is a performance boost by log filtering, and using log_level info instead of debug.

Disclaimer: I am not an official Substratum team member, just a community supporter. This write-up is interpretation of the Show & Tell meeting and data released from the team. None of this is to be taken or construed as investment-related or as financial advice.

Substratum Dev Team Show & Tell — 31-May-19

Show & Tell weekly sessions are an excellent insight into the challenges and triumphs facing the Substratum development team. Each week a set of slides is released and a group conference is held with dialogue and examples of work completed, tabled and planned on the horizon!

This week showed some exciting troubleshooting, and some major steps forward in both the monetization and neighborhood dynamics areas of Substratum Node.

With an air of battle hardened precision, B.J. Allmon lead developer begins the meeting by always outlining the Strategic Goals of v1.0.0 Substratum Node:

Priority 1 — Make the Internet Free (as in Freedom) and Fair

Priority 2 — Earn Cryptocurrency. Change the World

Tactical Focus Items are also listed and briefly mentioned as Work-In-Progress

Notably mentioned this week was how the team worked with Brian M, an amazing community node-running tester, who live tested a 5-node cluster with the developer team to troubleshoot. This led to 3 defect cards (the team uses Agile lean project management techniques) outlining fixes.

The team created an “Epic” story for RC1 critical fixes to correct some further routing bugs. These fixes are being implemented in next week or so.

Another minor version roll out is imminent (0.4.7)

RC1 is starting to be packaged and continuous work around this each and every day.

It was most exciting to hear comments about horizon planning for both Substratum Host and Cryptopay!

The actual Show and Tell segment takes up the bulk of each weekly meeting. Team Members take turns to ‘Tell’ the team and community of code commits or fixes, and also ‘Show’ on-screen changes or new implemented features to demonstrate their use.

Outlined below are the major points to digest from this week!

Creating a Sub Config File — Dan Wiebe:

Dev team implemented a ‘config file’ into Node software, because at the moment, the CLI (command line) startup command has a lot of parameters that need to be typed each time a user wants to run Substratum Node. (or integrate these parameters or settings into the environment)

Issue is, there could be prying eyes viewing a user’s command-line if typed each time. Not to mention it is cumbersome for testing and for users to simply start node on machines where their settings are not changing much.

Accountant scans & add delinquencies to ban — Dan Wiebe:

The Accountant module within SubNode keeps track of ‘who owes what, to who’ for routing, and from time to time refers to a receivables table of these values to ‘balance the books.’

Bills accumulate towards a threshold (based on a curve) where the bill is to be paid for consuming.

There is another threshold noted as a delinquency limit, and when these limits are exceeded, the accountant will add that consuming wallet to a ‘ban-list’ in memory, because their debt for consuming hasn’t been paid.

Based on this, a user using a wallet on the ban-list will not have their packets routed — meaning that their node will not be able to consume, unless they pay their dues, or consume across Node with another wallet with a positive balance.

Another upcoming change on this item will be that nodes with a consuming wallet on the ban-list will be removed from the ban-list when they have paid enough of what they owe.

Node Gossip not properly introducing 6th Node — Dan Wiebe:

Currently the node neighborhood is designed to have up to 5 immediate neighbors connected to each individual node.

A bug was detected through testing with Brian, where a 6th node was not being properly passed on to another nearby neighbor to attach to, when it tried to connect to a node that was ‘full’ (already had 5 connections). There was a discrepancy with passing on the newcomer node to another one nearby. The gossip protocol was not checking with the neighbors being passed to about whether the newcomer was already a connected neighbor to them. Due to this, new nodes joining were occasionally being passed back and forth and would not connect properly to neighbors nearby.

No more bootstrap nodes — Matt:

The bootstrap function has been deprecated and the code cleaned up to remove this. There were a few parameters and command references now removed.

Now nodes can connect freely with neighbors, by debut and passing gossip.

Removal of EntryDNS crashpoints — Dan:

During testing, there were Node crashes when an EntryDNS query failed trying to receive an invalid DNS request from a browser process. There was a fix put in so the Node doesn’t simply crash when this behavior occurs, it just gives an internal error to the EntryDNS module.

SHOW — Demo from BJ:

B.J. put the updated software GUI on-screen to show the changes. The updated SubNode GUI shows a much more streamlined interface. The forms only show the required fields to Serve and Consume, and there will be a ‘Config’ option in the Node menu to allow configuration of settings into the config file that was integrated.

Changing the Node status (between Off>Serving>Consuming etc) will retain the configuration that was inputted by the user.

Note — configuration settings only persist while the Node instance is running. It will be embedded in future versions.

Also in future version coming soon, there will be data shown in the UI like statistics of routing, earnings etc

Account Payment Detection — Matt:

This module scans the blockchain for the highest block number, and detect recent transactions to update the accountant module and payment statuses (keeping the books up to date!). Block number stored in database, so scanning can be done in forward fashion.

Over last 9 weeks, the team has seen constant progression, with each weekly iteration drawing a week closer on the countdown.

The team ‘just scraped through’ this week on the countdown, with the tech debt involved this week working on the 3 defect cards (it’s Brian’s fault lol).

“Team worked really hard, and getting stuff pushed through.” — BJ

This is a moving target — scope can change, people could be off sick etc!

“This week reflects the work we did with the testing community… the 29% and defect fixes…”

BJ explained that the team is now ‘playing whack-a-mole’ on remaining issues we don’t want in version 1.

57% capacity was on the ‘planned work’ side so that looks really good.

Guest Questions/Comments during meeting:

From Brian — the team ‘kicked butt’ with the bugs! Things looking good with 5-node and 6-node clusters in current tests! Routing works perfectly with HTTP & HTTPS!

Will continue to test in isolated test environment.

Had to delete the sqlite database (node-data.db) when restarting nodes and building new networks.

Steve Swing — Name and location of database (.db) file has stabilized so it may be prevalent, but is easy to work around now with that.

Overall seems like another weekly ‘notch on the belt’ for the project timelines, with a successful work-week accomplishing fixes and cards leading to v1 RC1 (Release Candidate 1). The sense of relief in the discussion was noticeable from the team, since it was a short work week in the USA, but the meeting was also quieter than usual as some regular attendees were not present.

Stay Tuned for the next update!

Disclaimer: I am not an official Substratum team member, just a community supporter. This write-up is interpretation of the Show & Tell meeting and data released from the team. None of this is to be taken or construed as investment-related or as financial advice.