Kill the radio (or die trying)

  • 202312.12

    Stamp protocol: a p2p identity system

    If you follow my work much, you’ll know I’m working on Basis, a project with the end goal of supporting a post-capitalist economy. I recently decided to move from the theory of the project into a more concrete implementation. But how? A few years back, I would have reached for blockchain tech but I’ve recently become increasingly suspicious of blockchains as a means for building distributed systems. Their reliance on global consensus means I have to “agree” that a transaction between a fisherman and a street vendor in a coastal town somewhere in Italy is valid…in truth, I don’t give a shit. Let them figure it out. It doesn’t scale and it never will. Blockchains are the ultimate rule-by-committee system where every decision has to be ratified by every participant. Kind of useless for anything beyond banks clearing transfers or dicks scamming idiots with NFTs.

    So the next choice is some kind of p2p system. If constructed properly, these systems can scale almost infinitely. However, control mechanisms need to be in place. If some actions in a p2p network protocol are not allowed, who makes sure they aren’t happening? Projects like Holochain try to solve this, and have made decent headway in doing so. In fact, I’ve long followed the Holochain project and been interested in using it for Basis or even Turtl.

    One thing that just bugs me about Holochain – and every other p2p system I’ve run into – is that your identity in the system is tied to a single cryptographic key. This key is you: and if the key is lost, “you” no longer exist in the system. This doesn’t end here, though. The “you” key is generated once on one of your devices and forever after lives just on that device with no built-in way to sync it to your other devices. This means there’s a different “you” on your phone, laptop, etc.

    Identity in p2p systems has always been an afterthought.

    Stamp was born

    I originally thought of Stamp as a way to cryptographically provide personal identity within a network. It’s much like PGP in that it works by using cryptographic keys as mechanisms for managing the identity. You can make claims, and others in the system can “stamp” (verify) your claims by signing them. This creates what’s known as a “web of trust.” If I trust Joe, and Joe stamps Jill’s identity, I can now extend a bit of trust to Jill.

    Stamp protocol logo

    Stamp differs from existing key-based identity systems in a few ways, however. It allows creating claims that can be automatically verified, such as the ownership of a domain or URL. It allows claims to be named, opening the door for discovery in other distributed systems (ie, maybe I have an “activitypub” URL claim that points to my Mastodon account).

    Another difference is identities in Stamp are not singular objects, but rather collections of “transactions” that are issued by the owner. These transactions link back to previous transactions, creating what’s known as a directed acyclic graph (DAG). This structure allows for auditability, versioning, and easy verification.

    Identity management

    Of all the differences between Stamp and other identity systems, the most important is that a Stamp identity is not controlled by a single key. Instead, identities have a set of “admin” keys that are assigned abilities using what are known as “policies.” Effectively, a policy is a set of permissions granted to a key or set of keys, allowing those keys to perform certain actions if they are used in the correct combinations.

    But why the complexity? Because it opens the door for another paradigm: group management of identities. Now instead of having to build a permissioning system around a singular key that allows someone in a group to create a signature on behalf of an identity, Stamp incorporates the permissioning system directly in the protocol. Policies can grant keys outside of the identity the ability to perform actions within the identity, so before Jill can sign a purchase order on behalf of the company, she needs to get approval from Ted in accounting.

    It doesn’t stop at group management, though! One of the most significant problems with existing identity systems is that because they are managed by a single key, if that key is lost then the identity is useless. Stamp solves this: the policy system allows you to specify a recovery mechanism for your identity. You can set up a policy that says “my spouse, brother, and any one of three friends can sign a key replacement transaction” and if you get the correct signatures, you regain access to your identity again. Not bad.

    Syncing and storage

    Instead of forcing an identity to live on one device and making someone run a bunch of ugly CLI commands in a terminal to move their key to another device, Stamp has built-in private identity syncing for your devices. Although not part of the core identity protocol, the project itself has a layer for keeping your identity synced between your devices.

    Included in this syncing, which is accomplished by setting up a p2p network, is public storage of published identities. So ditch those broken GPG key servers…from now on, you only have to look in one place for published identities, and if you’ve got an internet connection, you can look up an identity.

    Identity in p2p systems

    Ok so I’ve been blathering on and on about this great new identity system. What’s it good for, actually? Remember how I mentioned a Stamp identity is a collection of transactions? Well, turns out, it can issue transactions for things other than just ones that modify the identity.

    In fact, Stamp can issue any kind of custom transaction in the same envelope that lets any system that can speak “Stamp” verify that the transaction is valid (when I say “valid” I mean that the transaction is signed by the key(s) required to issue it via the policy system). This means that a Stamp identity can act as a multisig system for issuing transactions in other p2p systems. In effect, p2p systems no longer have to build a concept of identity because Stamp is handling it all for them.

    The ability to perform multisig external operations is instrumental for Basis because much of the protocol relies on the ability for groups of people to act as singular economic entities.

    What’s next?

    The Stamp core protocol is mostly complete at this point. I’m currently working on the network layer of the system: private syncing and publishing. Private syncing is being built in mind with other systems in mind: the next version of Turtl is going to be a p2p system and I’m trying to generalize things in Stamp’s networking layer enough to support Turtl’s need for person-to-person sharing.

    Basis has somewhat different requirements though: in effect, groups in Basis have cost allowances that they need to keep their costs under. If they make an order that increases their costs past their allowance, the transaction creating that order is invalid. How do we enforce this effectively? This is a topic of thought for me, and likely involves some sort of federated verification network within the larger p2p system. This is one of those places people reach for a blockchain, and rightly so, however I’d like Basis to be able to run more than 10 transactions a minute so an alternative is needed.

    Comments

  • 202004.10

    The property "ownership" myth (and other thoughts on socialization)

    It was getting dark, and starting to rain. You needed a place to stay. Not just for the night, but a place to call your own after living under other people’s roofs for so long. You’re tired of moving, tired of the journey. You want to put down roots.

    By strange coincidence, a stranger approaches you and says “See that house over there? I’ll sell you the house and the property it’s on for $50K.” You have the money, you’ve saved up for a long time. You agree.

    “There’s just one or two more things,” says the stranger. “You can’t make significant modifications to the property without asking me first.” It seems reasonable, so you agree. “Any large-scale projects you wish to do on the property must be cleared by me,” stated the stranger. “And lastly, at any point, if I ever need the property for something, I can take it back from you, and if I do this I will pay market value…but you cannot refuse.” It struck you as odd, but you agree and buy the property.

    Do you own it?

    Property ownership in the United States…

    …is a myth. Nobody owns (land/”real”) property. What people own is a title of use of property, but the property itself is jointly owned between cities, counties, states, and the nation. I’ve had this argument several times with many people, and none of them can reconcile “ownership” with the fact that:

    • You often need to get government approval to modify an existing structure on property
    • You need government approval for large-scale projects on property (building any kind of large structure, digging a mine, etc)
    • Most importantly, the government can take your property away from you via Eminent Domain whenever it deems it necessary (you being compensated is irrelevant to the fact that you have no choice)

    You do not own the property you “own” but rather own a title of use, and that title comes with various restrictions. If you can prove me wrong, please feel free to comment below, but to my understanding, if someone gives you something but has the ability to take it away from you at any point, you do not own it.

    It gets interesting with things like a civil suit in which being on the losing end means some amount of your assets being seized. Did you own those assets to begin with? It’s an interesting question, but ultimately I would say you did own those assets, but through some form of wrongdoing via your own action/inaction lost access to them, via some socially-mandated set of rules that come along with social or economic interaction with other people.

    That said, the fact that you can lose your property without wrongdoing and just because the government wants it, to me, points to the fact that the property was not yours to begin with.

    Why does this matter?

    This matters because given that property is not owned individually, it follows that property is socialized. In the context of a market system, it means that when you buy/sell property, you’re really just selling the title for use of the property, not the property itself.

    So when talking about socialization of productive property or housing, nothing regarding ownership changes materially: the ownership of property is already socialized! What changes is the mediation of the use of property. What this mediation looks like without markets and how it works is another very interesting but complex set of ideas (which are outside the scope of this post).

    However, the next time someone says that socialization of property is totalitarian or “doesn’t work” or some such nonsense, tell them that all property is already socialized and ownership is a myth.

    Comments

  • 201909.13

    The Five Components of Economic Systems: A discovery through explorations of Socialism

    As an opening note, I’d like to mention that while I came to the conclusions discussed in this post while researching what a “socialist mode of production” would look like, I’m attempting to the best of my abilities to not argue for one system or another. Rather, my goal is to share my findings on how economics can be broken down.

    It was December 2018. I had recently read through a forum discussion about Universal Basic Income, an idea I supported at the time. One of the comments mentioned how socialists view UBI as a band-aid fix.

    “Wait,” I thought. “Isn’t UBI a form of socialism?” I realized that even though my past 18 year old self had donned the label anarcho-communist, I didn’t actually know what that meant, other than “things should, like, be more equal.” I had left anarcho-communism behind in my 20s when I came to the realization that capitalism does work and became a supporter of it. But for some reason, when this last December rolled by, I connected with my old self again. You see, I have a problem. An addiction. I need to know how things work. It’s why I know a several forms of assembly language. It’s why I have a drawer full of deconstructed electronic components. I can remember my dad tell me as a kid: “Don’t take this apart!”. And now I needed to know what socialism meant; to deconstruct it and understand it. In our culture, the things we most attribute to socialism (state capitalism and welfare capitalism) have nothing to do with socialism. So what is it?

    Then it started. A strange manic episode which seemed to come from nowhere. For weeks, I was waking up after only four or five hours of sleep. I would open my eyes, the sky would still be dark, and I was exhausted, but my head was racing. I used my early mornings to my advantage: reading papers, researching economic theory, architecting systems, thinking through methods and incentives. I swam through ideas like I had woken up at the bottom of a lake and needed air: libertarian socialism, central planning, free association, market socialism, common ownership, democratic socialism, communism, syndicalism, social dividends, collectivist anarchism. Marx, Kropotkin, Cockshott. USSR, China, Cuba, South America, the Spanish Civil War. A raging torrent of information, history, and theory. I found a subreddit where people argue bitterly about economics. I talked with people and I fought with people, socialists and capitalists alike. I formed opinions. I talked more. I fought more. I asked questions. I reshaped all my opinions. Then I did it all again.

    The whole time, I was working on a project that would be the ultimate conclusion of my findings (which I will publish at a later date). This work has ushered me into a deeper understanding of how things are organized and how the pieces of economics all fit together.

    The five components of economic systems

    I ultimately found that there are five main components that describe an economic system. Because these are not apparently obvious, I thought it would be helpful to document my observations:

    • Ownership. This is the crux of the capitalism vs socialism debate. Capitalism necessitates private ownership of the means of production. Socialism is the social ownership thereof. This means capitalism simpler: anything can be privately owned. Socialism is a bit more complicated, because it makes a distinction between private property (factories and stuff) and personal property (your hat, your Roomba, your toothbrush). This distinction is lost on many people (aka the “you want to take my toothbrush” problem).
    • Organization. Is the economy completely decentralized and self-organizing? Or is there one entity that plans every input and output down to meticulous detail? It’s important to note that the former is attributed to capitalism, and the latter to socialism. This is mainly because most Marxist-Leninist states employed central planning for their economic system. But central planning is not socialism…social ownership of productive property is. It’s possible to have privately-owned production with heavy central planning (for instance, Nazi Germany). It’s possible, in theory, to have socially-owned productive instruments with a small (or no) government. We also have central planning in the United States (NASA, farm subsidies, etc).
    • Costs/valuation. Valuation can be done with prices guided by market forces, which is the mode we’re all used to because it’s how capitalism does things. However, there are more methods of valuation. There’s the Marxian value theory in which a thing is valued at the average amount of labor it took to build it. There is in-kind valuation, where all possible aspects of costs are tracked for a particular item (for instance, the labor that went into it, how much of each raw material, externalities if possible, etc). It’s important to note that cost and value are different: there is some intersection between the two, but value might ignore a cost like pollution, and cost might not take into account how something is valued to a particular individual.
    • Exchange. What medium do productive entities use to exchange? Money is the most prevalent one. There are other ideas, though, such as the free association of producers, where exchange happens freely…producers order things from other producers in order to meet some social need. If using central planning, exchange would look like an internal transfer might look at a large corporation.
    • Distribution. Who gets what? If using a money system, those with money buy what they want. If using gift economics, distribution might be a function of reputation. If labor vouchers, distribution is a function of hours worked (and possibly the social value of that labor). In some systems, distribution is need-based (“to each according to their needs”…I realize this is loosely defined). Distribution could be ration based as well: each person gets a daily allowance of resources they can consume. It’s important to note that the distribution mechanism is also what puts downward pressure on resource consumption and incentivizes higher levels of efficiency in the productive process: producers must make things in such a way that those who want those things can afford them.

    In theory, some of these parts can be swapped out for another. Also, few of these categories are binary: you can have some mix of public/private ownership, or some mix of decentralized/central planning. However, note that money can be used in a number of these economic components. This is a beautiful selling point of moneyed capitalism, but also what makes it hard to pick the above concepts apart: it’s simple. Anyone can own anything, and valuation/exchange/distribution are all based on money.

    Also, notice we’re missing the authoritarian aspect here completely. This is on purpose. Economics and authoritarianism are completely separate topics, as much as people like to lump them in together. It’s possible to have a totalitarian government with privately-owned production. It’s possible to have a tiny government with social ownership of production.

    Let’s clarify some ideas.

    Ownership

    I’d like to talk about the concept of ownership. I think it’s important to mention because it’s a deep topic of debate between proponents of socialism. What does it mean to have “common ownership over the means of production?” What even are the “means of production?”

    For the productive instruments to be socially owned, there are a number of different ideas. First, the most common understanding is state ownership. This is tricky, though. If the state is representative of the people in it, then by the state owning the means of production, the people do as well. However, if a state is authoritarian such that the citizens have very little control, then the productive instruments are not held in common, but rather owned by oligarchs. This is in contrast with more anarchist ideas, in which ownership of means of production is directly social, as in, not proxied through a state (or perhaps, in the libertarian case, owned by a municipality) and use thereof is determined cooperatively.

    Then, there’s the “means of production” themselves. There are as many definitions of this as there are socialists, but one common theme I’ve seen over and over would be large capital assets: raw materials/resources, productive land (farms), factories, warehouses, office buildings, air/sea ports, large machinery, and intellectual property. Essentially, anything that’s used exclusively for productive purposes would be a target for social ownership. It would be possible to socialize laptops and toothbrushes and things of that nature, but it likely would not be very effective at all.

    Lastly, let’s talk companies themselves. Socialism necessitates abolishing private ownership of companies. What does this mean, though? Does the state own all companies? Do the employees? The answer is it depends. In some models (such as central planning), yes, the state would own, or at least orchestrate, all companies. In other systems, companies would be self-determined and would be state-agnostic. It’s important to note that just because the means of production are socialized doesn’t mean that the companies using those means of production need to be socialized as well.

    Organization

    Another important topic is how the economy is organized. It’s often thought that decentralized organization cannot work without prices/profits. I’d like to demonstrate that as theoretically false.

    If you have a transparent system that allows orders between productive entities (for instance, some kind of blockchain), demand can be measured directly by looking at the order list. Instead of, then, using price signals to determine supply levels, supply can be matched directly to demand.

    In other words, money is a method of exchange between productive entities, but it’s not the only method of exchange. Organization is as much about communication of needs as it is about a means of exchange: if needs can be communicated directly, then distributed production can be enabled, no matter what valuation system (or system of exchange) is used.

    That wraps things up. I hope reading this was as illuminating to you as it was for me to discover it.

    Comments

  • 201904.02

    The mystification of pricing

    Pricing is an important economic tool. How much things cost, and how much things should cost, has been a subject of debate between economists for a long time. However, there’s one part of pricing I’d like to dissect. It’s the aspect of mystification.

    When I buy a product, it sends a signal. It’s essentially an economic feedback mechanism that says “I value this product as much or more than the price that it is being sold for.” This signal is transmitted back to the producer (sometimes over many hops of the economic network because usually you buy a product from a distributor, not directly from the manufacturer). However, the producer does not know why I bought it.

    The price takes into account many different factors: the inputs to production required to make the product, the wages of the people producing it, the marketing budget to convince people to buy it, the costs of compliance with regulations, the need to pay back interest on loans acquired for startup capital, shipping costs, etc etc. There are likely thousands upon thousands of factors that are bundled into the price of a product.

    The problem is, we don’t know what they are. When I buy something, I am saying “I like this for some reason” and that’s the only economic signal the producer receives. I can call them and say “Hey, I love your peanut butter because it tastes better than others” and companies will certainly encourage this, but there is no economic feedback mechanism other than the purchase itself.

    So, if labor costs $5, the inputs to production cost $3, marketing is $1, and the owner of the company wants a profit of $3, that’s 5 + 3 + 1 + 3 = 12. Now, as a consumer, how do I tell how much the labor cost for this product? Or how much profit was made? It’s impossible. You can only find one by knowing all of the others.

    Why is this a problem? It’s a problem because people encourage voting with capital. They encourage spending based on beliefs. But, there’s no economic feedback mechanism in place to vote for just one of the factors of the price of a product.

    Imagine if you had deconstructed pricing for every product. Instead of $5.99 it had $0.03 in environmental compliance, $0.50 in marketing, $1.00 in wages (with a breakdown of labor-hours averaged per-worker), $0.60 in productive inputs (and each input had a deconstructed price as well), $1.67 in equipment maintenance, $0.98 for reinvestment in the company, and $1.21 in profit for the owners.

    Now, pricing is demystified. You can buy a product produced by people who are paid well, or buy products that are made from products made by people who are paid well. You can buy a product that spends more in order to care for the environment. You can buy products that value profit less.

    Would this not be a a requirement of voting with your dollar?

    This thought experiment came out of discussions I’ve have around an economic model I’m working on. One of the ideas I’m interested in is wages being decided democratically based on a set of attributes: social need, skill/training required, stress level, danger, etc. Essentially, each attribute would be a value between 0 and 1, and the average of all the attribute values multiplied by a maximum wage would be someone’s hourly wage. A productivity attribute could be applied to each person such that their occupational wage goes from being fixed to being a range instead, and this productivity rate would be decided by the worker (if self-employed) or the members of their company. The wage attributes would be decided regionally (per-city/county).

    In essence, you give every industry-occupation pair a score and use it to determine the range of wages a person can have. After that, you let the pricing system take over. The purpose here is for society to make a conscious decision on “what this job is worth.” My theory is that the economy would be shaped very differently if people were given a direct choice. Want smarter children? Raise the need for the education industry and for the teacher occupation. Too many tech dorks on scooters? Lower the social need for computer programmers. Making wages a conscious decision of the members of the community allows them to shape their own economy, without having to deal with the wildly inaccurate pricing system and the feedback loops of the market.

    Should a pro football player be paid 500x more than a teacher? I’m willing to bet that if you asked every single person in the United States this question, the answer would overwhelmingly be “no.” Why, then, does the market enforce these patterns?

    After discussions, this idea of democratically setting wages was met with some resistance. It was argued that the market already does allow democratization of wages. People do vote for salaries already, and they do it via the products and services they buy. However, after some thinking, I realized this isn’t true. People certainly enforce economic patterns via their spending, but the truth is that, even if they did want to alter their spending to shape the economy according to their desires, they simply do not have the information they need to do so.

    This is because of the mystification of pricing.

    UPDATE - Feb 12, 2020

    It’s important to note that I’ve since abandoned the idea of democratically-decided wages. While the idea was made in good faith, I couldn’t solve for things like

    • On call/emergency hour rate hikes. Sure, my day rate is $100/hr, but if you need me at 3am on the weekend, it’s $200/hr
    • Scale. What does a musician make for performing gigs? What if they are playing shows for 1000+ people?
    • Distributed decision making. My skills might be highly valued, and forcing a wage on someone when their time is worth more (both to them and the other party) is needlessly authoritarian. Shouldn’t the level of stress, danger, need, training, etc be decided by me and not the general population, who knows jack shit about my job?

    I believe all of the problems that might be solved by democratic wages would actually be better handled by democratic workspaces (worker-owned companies) and free association. Not only that, but an excessive amount of beureaucracy is removed (no need to vote just to set wages). While I still support a profitless economy, it should work within the confines of workers being able to negotiate their own wages.

    The other stuff I wrote about pricing in the post still stands, however.

    Comments

  • 201710.27

    How to really save Net Neutrality

    Net Neutrality is a hot topic in the United States. On the one hand, you have people claiming that government should stay out of the internet. On the other hand, you have people claiming that the internet is much too important to leave in the hands of a few immense telecoms.

    What is Net Neutrality?

    Net Neutrality is fairly simple. There are many people who will tell you it’s the “government controlling the internet” or “Obamacare for the internet.” This is useless propraganda.

    Net Neutrality is the idea that all traffic is treated the same, regardless of its source or destination. From the college student looking at cat photos to the government contractor submitting plans for a new missle prototype: if it’s going over the internet, the packets carrying the information are treated the same.

    That’s it.

    Why is this important? Well, Comcast has a service, Stream, that competes with Netflix. There are two ways to make money by sidestepping Net Neutrality: Comcast can tell Netflix “pay us a buttload of money every year or we’ll make sure your little streaming service is unusably slow for our customers.” Netflix now has to pony up or lose a large segment of paying customers. On the flipside, Comcast can throttle Netflix and start marketing their Stream service so their customers will more naturally flock away from Netflix.

    That’s one example. There are a number of things telecoms can do if Net Neutrality is not enforced:

    • Throttle competing services
    • Sell tiered internet plans:
      • $50 for Basic (Google, Facebook, Twitter)
      • $70 for Premium (Basic + CSS, Fox, and 12 other news websites)
      • $120 for Platinum (all websites)
    • Block websites outright.

    Looks bleak.

    The principals of Net Neutrality are clearly pro-consumer. Having equal access to all information on the internet without any kind of gatekeeper forcing you into various acceptable pathways is what our country is about: open exchange of ideas.

    So why don’t telecoms enforce Net Neutrality?

    The problem(s) with the internet in the US

    Most big telecoms abhor Net Neutrality as they see it as a barrier to their profit margins. They don’t want to treat traffic equally, going so far as to say it hinders their free speech.

    But won’t the free market solve this problem? The answer is “not really.” The free market, as it exists in its current climate, has solved this problem. We are already looking at the solution: A handful of large players, dividing up service areas on gentleman’s agreements, effectively self-enforcing a one-company-per-area monopoly for any given town. In essense, there is no choice in ISP, other than to move to another town.

    Another problem is that given that these companies act as gateways to the world’s online information, they are given cash infusions by various governments in order to expand their networks. These expansions either don’t happen at all or if they do, are miniscule compared to the promises made.

    In the cases where people decide their town should build fiber lines that are truly owned by the public, the telecoms will file suit and run propaganda campaigns in the towns in order to fight what is essentially free-market competition (with a municipality entering the market as a competitor).

    Summary:

    • Telecoms are monopolies in the US. There is no “choice” in most locations.
    • Telecoms block any competitive choice through collusion or through passing legislation blocking municipal broadband.
    • The people of the US have invested billions in telecom infrastructure, but are told we have no choice when deciding the flow of information through the networks we’ve invested in.

    The solution: Municipal fiber

    Net Neutrality has been ping-ponging in the FCC for a while now. Things were looking good with Wheeler in charge. Now things look dark with Pai. If there is a decisive win either way, the battle will move to congress. Telecoms are pouring money by the truckload into their anti-Net Neutrality campaigns. At the same time there are a vocal group of people fighting to protect NN.

    The battle will rage tirelessly for years to come unless we change our methods.

    We need to move the battle out of the federal government and into local municipalities. We need to crush the telecoms with public infrastructure. We need fiber in our towns, and LTE towers in our rural areas, all publically owned and operated. Then we can rent out the infrastructure to whoever wants to compete on it.

    This creates a level playing field for true competition, while putting the supporting infrastructure where it belongs: in the hands of the public. We have municipal roads. We have municipal water. We have, in many places, municipal power.

    It’s time for municipal fiber.

    This will end the stranglehold telecoms have on our information. It allows the free market to solve the issue of Net Neutrality through competition, making it something that no longer needs regulatory protection.

    The Net Neutrality activists win. The free market fundamentalists win. The only losers are the entrenched powers that are squeezing your wallet while tightening their grip on the flow of information.

    Talk to your city/county/state representatives about municipal fiber.

    Comments

  • 201702.01

    Debug comments (or how to save your sanity using git)

    A lot of times when I’m programming, I need to write a few lines of code that test what I’m working on. This can be a top-level function call, a few log entries, etc. Much to my dismay, I tended to end up with this debug code committed to git.

    I decided I wasn’t going to take it anymore.

    Git’s pre-commit hook to the rescue

    Now, whenever I add one of these lines, I mark it with a special comment:

    console.log('the value is: ', val);     // gee, sure hope i don't commit this

    becomes

    // DEBUG: remove this log entry
console.log('the value is: ', val);

    Then in my pre-commit hook, I symlink a script that checks for DEBUG comments in various languages for that repo:

    #!/bin/sh

DEBUG=""
function add_to_debug {
	filetype=$1
	comment=$2

	next=$(
		git diff \
			--cached \
			--name-only \
			-G"${comment}[ ]*DEBUG" \
			-- "*.${filetype}"
	)
	DEBUG="${DEBUG}${next}"
}

add_to_debug 'js' '//'
add_to_debug 'rs' '//'
add_to_debug 'html' '<!--'
add_to_debug 'lisp' ';'
add_to_debug 'sh' '#'
add_to_debug 'hbs' '{{!--'

if [ "${DEBUG}" != "" ]; then
	echo
	echo "Please address the DEBUG comments in following files before committing:"
	echo
	echo "${DEBUG}" | sed 's/^/  /'
	exit 1
fi

    Using this, trying to commit any code that has DEBUG comments will fail with the output:

    Please address the DEBUG comments in following files before committing:

  user.js

    This forces going back in and cleaning up your code before committing it. Wicked.

    Get it yourself

    Grab the pre-commit hook off my Github to END THE SUFFERING and stop committing your debug code.

    Comments

  • 201606.05

    Ansible: included handlers not running in v2.x

    I’ll keep this short. I recently installed Ansible 2.0 to manage the Turtl servers. However, once I ran some of the roles I used in the old version, my handlers were not running.

    For instance:

    # roles/rethinkdb/tasks/main.yml

- name: copy rethinkdb monitrc file
  template: src=monit.j2 dest=/etc/monit.d/rethinkdb
  notify: restart monit
    # roles/rethinkdb/handlers/main.yml

- include roles/monit/handlers/main.yml
    # roles/monit/handlers/main.yml

- name: restart monit
  command: /etc/rc.d/rc.monit restart

    Note that in Ansible <= 1.8, when the monitrc file gets copied over, it would run the restart monit handler. In 2.0, no such luck.

    The fix

    I found this github discussion which led to this google groups post which says to put this in ansible.cfg:

    [defaults]
...
task_includes_static = yes
handler_includes_static = yes

    This makes includes pre-process instead of being loaded dynamically. I don’t really know what that means but I do know it fixed the issue. It breaks looping, but I don’t even use any loops in ansible tasks, so

    Do whatever you want, you little weasel. I don't care. I DON'T CARE.

    Comments

  • 201511.22

    SSH public key fix

    So once in a while I’ll run into a problem where I can log into a server via SSH as one user via public key, and taking the authorized_keys keys and dumping it into another user’s .ssh/ folder doesn’t work.

    There are a few things you can try.

    Permissions

    Try this:

    chmod 0700 .ssh/
chmod 0600 .ssh/authorized_keys
sudo chown -R myuser:mygroup .ssh/

    That should fix it 99% of the time.

    Locked account

    Tonight I had an issue where the permissions were all perfect…checked, double checked, and yes they were fine.

    So after poking at it for an hour (instead of smartly checking the logs) I decided to check the logs. I saw this error:

    Nov 23 05:26:46 localhost sshd[1146]: User deploy not allowed because account is locked
Nov 23 05:26:46 localhost sshd[1146]: input_userauth_request: invalid user deploy [preauth]

    Huh? I looked it up, and apparently an account can become locked if its password is too short or insecure. So I did

    sudo passwd deploy

    Changed the password to something longer, and it worked!

    Have any more tips on fixing SSH login issues? Let us know in the comments below.

    Comments

  • 201509.05

    Nginx returns error on file upload

    I love Nginx and have never had a problem with it. Until now.

    Turtl, the private Evernote alternative, allows uploading files securely. However, after switching to a new server on Linode, uploads broke for files over 10K. The server was returning a 404.

    I finally managed to reproduce the problem in cURL, and to my surprise, the requests were getting stopped by Nginx. All other requests were going through fine, and the error only happened when uploading a file of 10240 bytes or more.

    First thing I though was that Nginx v1.8.0 had a bug. Nobody on the internet seemed to have this problem. So I installed v1.9.4. Now the server returned a 500 error instead of a 404. Still no answer to why.

    I finally found it: playing with client_body_buffer_size seemed to change the threshold for which files would trigger the error and which wouldn’t, but ultimately the error was still there. Then I read about how Nginx uses temporary files to store body data. I checked that folder (in my case /var/lib/nginx/client_body) and the folder was writeable by the nginx user, however the parent folder /var/lib/nginx was owned by root:root and was set to 0700. I set /var/lib/nginx to be readable/writable by user nginx, and it all started working.

    Check your permissions

    So, check your folder permissions. Nginx wasn’t returning any useful errors (first a 404, which I’m assuming was a bug fixed in a later version) then a 500 error. It’s important to note that after switching to v1.9.4, the Permission Denied error did show up in the error log, but at that point I had already decided the logs were useless (v1.8.0 silently ignored the problem).

    Another problem

    This is an edit! Shortly after I applied the above fix, I started getting another error. My backend was getting the requests, but the entire request was being buffered by Nginx before being proxied. This is annoying to me because the backend is async and is made to stream large uploads.

    After some research, I found the fix (I put this in the backend proxy’s location block:

    proxy_request_buffering off;

    This tells Nginx to just stream the request to the backend (exactly what I want).

    Comments

  • 201507.29

    Turtl's new syncing architecture

    For those of you just joining us, I’m working on an app called Turtl, a secure Evernote alternative. Turtl is an open-source note taking app with client-side encryption which also allows private collaboration. Think like a private Evernote with a self-hosted option (sorry, no OCR yet =]).

    Turtl’s version 0.5 (the current version) has syncing, but it was never designed to support offline mode, and requires clients to be online to use Turtl. The newest upcoming release supports fully offline mode (except for a few things like login, password changes, etc). This post will attempt to describe how syncing in the new version of Turtl works.

    Let’s jump right in.

    Client IDs (or the “cid”)

    Each object having a globally unique ID that can be client-generated makes syncing painless. We do this using a few methods, some of which are actually borrowed from MongoDB’s Object ID schema.

    Every client that runs the Turtl app creates and saves a client hash if it doesn’t have one. This hash is a SHA256 hash of some (cryptographically secure) random data (current time + random uuid).

    This client hash is then baked into every id of every object created from then on. Turtl uses the composer.js framework (somewhat similar to Backbone) which gives every object a unique ID (“cid”) when created. Turtl replaces Composer’s cid generator with its own that creates IDs like so:

    12 bytes hex timestamp | 64 bytes client hash | 4 bytes hex counter

    For example, the cid

    014edc2d6580b57a77385cbd40673483b27964658af1204fcf3b7b859adfcb90f8b8955215970012

    breaks down as:

     timestamp    client hash                                                      counter
------------|----------------------------------------------------------------|--------
014edc2d6580 b57a77385cbd40673483b27964658af1204fcf3b7b859adfcb90f8b895521597 0012
 |                                    |                                        |
 |- 1438213039488                     |- unique hash                           |- 18

    The timestamp is a new Date().getTime() value (with leading 0s to support longer times eventually). The client hash we already went over, and the counter is a value tracked in-memory that increments each time a cid is generated. The counter has a max value of 65535, meaning that the only way a client can produce a duplicate cid is by creating 65,535,001 objects in one second. We have some devoted users, but even for them creating 65M notes in a second would be difficult.

    So, the timestamp, client hash, and counter ensure that each cid created is unique not just to the client, but globally within the app as well (unless two clients create the same client hash somehow, but this is implausible).

    What this means is that we can create objects endlessly in any client, each with a unique cid, use those cids as primary keys in our database, and never have a collision.

    This is important because we can create data in the client, and not need server intervention or creation of IDs. A client can be offline for two weeks and then sync all of its changes the next time it connects without problems and without needing a server to validate its object’s IDs.

    Using this scheme for generating client-side IDs has not only made offline mode possible, but has greatly simplified the syncing codebase in general. Also, having a timestamp at the beginning of the cid makes it sortable by order of creation, a nice perk.

    Queuing and bulk syncing

    Let’s say you add a note in Turtl. First, the note data is encrypted (serialized). The result of that encryption is shoved into the local DB (IndexedDB) and the encrypted note data is also saved into an outgoing sync table (also IndexedDB). The sync system is alerted “hey, there are outgoing changes in the sync table” and if, after a short period, no more outgoing sync events are triggered, the sync system takes all pending outgoing sync records and sends them to a bulk sync API endpoint (in order).

    The API processes each one, going down the list of items and updating the changed data. It’s important to note that Turtl doesn’t support deltas! It only passes full objects, and replaces those objects when any one piece has changed.

    For each successful outgoing sync item that the API processes, it returns a success entry in the response, with the corresponding local outgoing sync ID (which was passed in). This allows the client to say “this one succeeded, remove it from the outgoing sync table” on a granular basis, retrying entries that failed automatically on the next outgoing sync.

    Here’s an example of a sync sent to the API:

    [
    {id: 3, type: 'note', action: 'add', data: { <encrypted note data> }}
]

    and a response:

    {
    success: [
        {id: 3, sync_ids: ['5c219', '5c218']}
    ]
}

    We can see that sync item “3” was successfully updated in the API, which allows us to remove that entry from our local outgoing sync table. The API also returns server-side generate sync IDs for the records it creates in its syncing log. We use these IDs passed back to ignore incoming changes from the API when incoming syncs come in later so we don’t double-apply data changes.

    Why not use deltas?

    Wouldn’t it be better to pass diffs/deltas around than full objects? If two people edit the same note in a shared board at the same time, then the last-write-wins architecture would overwrite data!

    Yes, diffs would be wonderful. However, consider this: at some point, an object would be an original, and a set of diffs. It would have to be collapsed back into the main object, and because the main object and the diffs would be client-encrypted, the server has no way of doing this.

    What this means is that the clients would not only have to sync notes/boards/etc but also the diffs for all those objects, and collapse the diffs into the main object then save the full object back to the server.

    To be clear, this is entirely possible. However, I’d much rather get the whole-object syncing working perfectly before adding additional complexity of diff collapsing as well.

    Polling for changes

    Whenever data changes in the API, a log entry is created in the API’s “sync” table, describing what was changed and who it affects. This is also the place where, in the future, we might store diffs/deltas for changes.

    When the client asks for changes, at does so using a sequential ID, saying “hey, get me everything affecting my profile that happened after <last sync id>”.

    The client uses long-polling to check for incoming changes (either to one’s own profile or to shared resources). This means that the API call used holds the connection open until either a) a certain amount of time passes or b) new sync records come in.

    The API uses RethinkDB’s changefeeds to detect new data by watching the API’s sync table. This means that changes coming in are very fast (usually within a second of being logged in the API). RethinkDB’s changefeeds are terrific, and eliminate the need to poll your database endlessly. They collapse changes up to one second, meaning it doesn’t return immediately after a new sync record comes in, it waits a second for more records. This is mainly because syncs happen in bulk and it’s easier to wait a bit for a few of them than make five API calls.

    For each sync record that comes in, it’s linked against the actual data stored in the corresponding table (so a sync record describing an edited note will pull out that note, in its current form, from the “notes” table). Each sync record is then handed back to the client, in order of occurence, so it can be applied to the local profile.

    The result is that changes to a local profile are applied to all connected clients within a few seconds. This also works for shared boards, which are included in the sync record searches when polling for changes.

    File handling

    Files are synced separately from everything else. This is mainly because they can’t just be shoved into the incoming/outgoing sync records due to their potential size.

    Instead, the following happens:

    Outgoing syncs (client -> API)

    Then a new file is attached to a note and saved, a “file” sync item is created and passed into the ougoing sync queue without the content body. Keep in mind that at this point, the file contents are already safe (in encrypted binary form) in the files table of the local DB. The sync system notices the outgoing file sync record (sans file body) and pulls it aside. Once the normal sync has completed, the sync system adds the file record(s) it found to a file upload queue (after which the outgoing “file” sync record is removed). The upload queue (using Hustle) grabs the encrypted file contents from the local files table uploads it to the API’s attachement endpoint.

    Attaching a file to a note creates a “file” sync record in the API, which alerts clients that there’s a file change on that note they should download.

    It’s important to note that file uploads happen after all other syncs in that bulk request are handled, which means that the note will always exist before the file even starts uploading.

    Encrypted file contents are stored on S3.

    Incoming syncs (API -> client)

    When the client sees an incoming “file” sync come through, much like with outgoing file syncs, it pulls the record aside and adds it to a file download queue instead of processing it normally. The download queue grabs the file via the note attachment API call and, once downloaded, saves it into the local files database table.

    After this is all done, if the note that the file is attached to is in memory (decrypted and in the user’s profile) it is notified of the new file contents and will re-render itself. In the case of an image attachment, a preview is generated and displayed via a Blob URL.

    What’s not in offline mode?

    All actions work in offline mode, except for a few that require server approval:

    • login (requires checking your auth against the API’s auth database)
    • joining (creating an account)
    • creating a persona (requires a connection to see if the email is already taken)
    • changing your password
    • deleting your account

    What’s next?

    It’s worth mentioning that after v0.6 launches (which will include an Android app), there will be a “Sync” interface in the app that shows you what’s waiting to be synced out, as well as file uploads/downloads that are active/pending.

    For now, you’ll just have to trust that things are working ok in the background while I find the time to build such an interface =].

    Comments

  • 201507.26

    Squeezebox setup without the remote/controller

    My dad recently gave me a Squeezebox for a present after he’d upgraded his home audio system. I was grateful but ultimately stumped on how to set it up. I read a bunch online about setting it up with the remote it comes with. Oh wait. Mine doesn’t have a remote.

    Let the fun begin.

    Connecting to the Squeezebox

    This is harder than it sounds. Initially, I tried wiring it into my router and seeing if it could see it. It could not. This was a WRT54G with Tomato firmware. Maybe the setups just weren’t compatible or something ridiculous like that.

    So I tried another way I found after poking around a lot: the Squeezebox has a build in wireless SSID that you can connect to in ad-hoc mode (after holding the only button for > 6 seconds and it enters reset/config mode).

    However, doing this is finicky and had me tearing my hair out. Ultimately, I got my Windows machine connected to it. When it connects, it gives your machine an IP in the 192.168.1.100-254 range. If it gives you a 169.xxx.xxx.xxx address, it’s game over. Try restarting your machine. Try resetting the Squeezebox. Try a rain dance while wearing a tribal loincloth. You just might get that IP.

    I recently bought a new router (Buffalo, w/ DD-WRT) and plugged the reset-mode Squeezebox into it (via LAN) and was able to connect to it instantly, so try that first and only go the ad-hoc wireless route if you absolutely have to.

    Talking to the Squeezebox

    The Net-UDAP software is amazing and wonderful. I don’t know what it does under the hood, but it lets you talk to your Squeezebox should you finally get connected to it in some capacity.

    Unless you’re on Windows. Yes, I know, it supposedly works on Windows but just didn’t find the Squeezebox with running discover.

    My only solution was to spin up a linux VM with a bridged network adapter and run Net-UDAP there instead. Worked flawlessly. Hopefully you have a linux box laying around, or maybe it will just work for you in Windows. Try the Windows perl binary instead of cygwin’s perl.

    Anyway, once you’ve got everything connected, you run the Net-UDAP like so:

    cd /path/to/net-udap
./scripts/udap_shell.pl -a 192.168.0.106

    Note that the 192.168.0.106 is the address for the machine you’re running the shell on, not the Squeezebox itself.

    You should get a prompt:

    UDAP>

    Now run the discover command:

    UDAP> discover
info: *** Broadcasting adv_discovery message to MAC address 00:00:00:00:00:00 on 255.255.255.255
info:   adv_discovery response received from 69:69:69:69:69:69
info: *** Broadcasting get_ip message to MAC address 69:69:69:69:69:69 on 255.255.255.255
info:   get_ip response received from 69:69:69:69:69:69
info: *** Broadcasting get_data message to MAC address 69:69:69:69:69:69 on 255.255.255.255
info:   get_data response received from 69:69:69:69:69:69

    Hopefully you get output like that. If you get empty output, see “Connecting to the Squeezebox” =[.

    Setup

    Once that finicky little bastard is discovered, run list:

    UDAP> list  
 #    MAC Address    Type       Status
== ================= ========== ===============
 1 69:69:69:69:69:69 squeezebox init

    You can see it has an ID of 1 so we do:

    UDAP> conf 1

    Your prompt will now change, and you’re in config mode:

    UDAP [1] (squeezebox 696969)>

    Now you can connect it to your network via wireless by setting values into its config. This will differ network to network, but here are the commands I run to get things working on a network with WPA2-PSK TKIP+AES:

    set hostname=jammy interface=0 lan_gateway=192.168.0.1 lan_ip_mode=1 primary_dns=192.168.0.1
set wireless_SSID='my network SSID' wireless_wpa_mode=2 wireless_wpa_cipher=3 wireless_keylen=0 wireless_mode=0 wireless_region_id=4 wireless_wpa_on=1 wireless_wpa_psk='WPA passwordddd' wireless_channel=9
set server_address=192.168.0.113

    For a list of the fields and what they mean, type fields:

    UDAP [1] (squeezebox 696969)> fields
             bridging: Use device as a wireless bridge (not sure about this)
             hostname: Device hostname (is this set automatically?)
            interface: 0 - wireless, 1 - wired (is set to 128 after factory reset)
          lan_gateway: IP address of default network gateway, (e.g. 192.168.1.1)
          lan_ip_mode: 0 - Use static IP details, 1 - use DHCP to discover IP details
  lan_network_address: IP address of device, (e.g. 192.168.1.10)
      lan_subnet_mask: Subnet mask of local network, (e.g. 255.255.255.0)
          primary_dns: IP address of primary DNS server
        secondary_dns: IP address of secondary DNS server
       server_address: IP address of currently active server (either Squeezenetwork or local server
squeezecenter_address: IP address of local Squeezecenter server
   squeezecenter_name: Name of local Squeezecenter server (???)
        wireless_SSID: Wireless network name
     wireless_channel: Wireless channel (used by AdHoc mode???)
      wireless_keylen: Length of wireless key, (0 - 64-bit, 1 - 128-bit)
        wireless_mode: 0 - Infrastructure, 1 - Ad Hoc
   wireless_region_id: 4 - US, 6 - CA, 7 - AU, 13 - FR, 14 - EU, 16 - JP, 21 - TW, 23 - CH
   wireless_wep_key_0: WEP Key 0 - enter in hex
   wireless_wep_key_1: WEP Key 1 - enter in hex
   wireless_wep_key_2: WEP Key 2 - enter in hex
   wireless_wep_key_3: WEP Key 3 - enter in hex
      wireless_wep_on: 0 - WEP Off, 1 - WEP On
  wireless_wpa_cipher: 1 - TKIP, 2 - AES, 3 - TKIP & AES
    wireless_wpa_mode: 1 - WPA, 2 - WPA2
      wireless_wpa_on: 0 - WPA Off, 1 - WPA On
     wireless_wpa_psk: WPA Public Shared Key

    To see the values already set, run list when in config mode.

    Great, so you’ve set up all your network values and are confident that you’ve done it all right the first time. Good for you. Now you can run save_data:

    UDAP [1] (squeezebox 696969)> save_data
info: *** Broadcasting set_data message to MAC address 69:69:69:69:69:69 on 255.255.255.255
ucp_method set_data callback not implemented yet at /path/to/udap/../src/Net-UDAP/lib/Net/UDAP.pm line 292.
Raw msg:
          00 01 02 03 04 05 06 07 - 08 09 0A 0B 0C 0D 0E 0F  0123456789ABCDEF

00000000  00 02 00 00 00 00 00 00 - 00 01 00 04 20 16 5A 05  ............ .Z.
00000010  00 01 C0 01 00 00 01 00 - 01 00 06 00 1A           .............
info:   set_data response received from 69:69:69:69:69:69

    Make sure save_data returns a response similar to this. If it doesn’t, run it again. In fact, run it again anyway. Run it again…and again…and again.

    Great, now run reset to finalize everything:

    UDAP [1] (squeezebox 696969)> reset
info: *** Broadcasting reset message to MAC address 69:69:69:69:69:69 on 255.255.255.255
ucp_method reset callback not implemented yet at /path/to/udap/../src/Net-UDAP/lib/Net/UDAP.pm line 292.
Raw msg:
          00 01 02 03 04 05 06 07 - 08 09 0A 0B 0C 0D 0E 0F  0123456789ABCDEF

00000000  00 02 00 00 00 00 00 00 - 00 01 00 04 20 16 5A 05  ............ .Z.
00000010  00 01 C0 01 00 00 01 00 - 01 00 04                 ...........
info:   reset response received from 69:69:69:69:69:69

    All done. Now don’t ever change your network setup ever again or you’ll have to deal with this shit again. Or just get a fucking remote…

    Comments

  • 201507.21

    Hackernews: a typical day

    Comments

  • 201507.18

    Switching to Jekyll

    I’ve decided to get rid of Wordpress that was on blog.killtheradio.net as well as the PHP site at killtheradio.net and combine both into a Jekyll blog on the http://killtheradio.net/ domain.

    Moving to Jekyll from Wordpress took a few days, but I got all my posts moved, edited to fix formatting errors, and switching all discussions to use Disqus (and of course imported the old comments).

    This site now works on mobile devices as well.

    There are a few reasons for all this, but mainly I’ve been intrigued by the idea of static site generators for a while now and wanted to try it out. Also, as time went on, I grew to desipise Wordpress, including all the idiotic security vulnerabilities I suffered through week after week. It’s a slapped-together platform, and the plugins for it are even worse.

    There’s a certain thrill to authoring and publishing new content using only the command line.

    Comments

  • 201501.26

    Node.js and Cygwin: Unknown system errno 203

    A recent Cygwin upgrade left me ripping my hair out, because none of my npm or grunt commands would work. They spat out the error

    Unknown system errno 203

    Helpful, right?

    The fix

    In your Cygwin ~/.bash_profile file (create it and chmod 755 .bash_profile if it doesn't exist):

    export TMP=/tmp
export TEMP=$TMP

    This did the trick for me. Special thanks to this github comment.

    Comments

  • 201409.18

    Sudoers syntax error for specific commands

    This will be short but sweet. When deploying some new servers today, I ran into a problem where no matter what, sudo bitched about syntax errors in my sudoers file. I tried a bunch of different options/whitespace tweaks/etc and nothing worked.

    deploy ALL= NOPASSWD: monit restart my-app

    Looks fine right? Nope.

    Use absolute paths

    This fixed it:

    deploy ALL= NOPASSWD: /usr/bin/monit restart my-app

    Everyone in the world's advice is to "just use visudo" but I couldn't find any info on what was actually causing the syntax error. Hopefully this helps a few lost souls.

    Comments

  • 201407.21

    Composer.js v1.0 released

    The Composer.js MVC framework has just released version 1.0! Note that this is a near drop-in replacement for Composer v0.1.x.

    There are some exciting changes in this release:

    • Composer no longer requires Mootools... jQuery can be used as a DOM backend instead. In fact, it really only needs the selector libraries from Moo/jQuery (Slick/Sizzle) and can use those directly. This means you can now use Composer in jQuery applications.
    • Controllers now have awareness of more common patterns than before. For instance, controllers can now keep track of sub-controllers as well as automatically manage bindings to other objects. This frees you up to focus on building your app instead of hand-writing boilerplate cleanup code (or worse, having rogue objects and events making your app buggy).
    • The ever-popular RelationalModel and FilterCollection are now included by default, fully documented, and considered stable.
    • New class structures in the framework expose useful objects, such as Composer.Class which gives you a class structure to build on, or Composer.Event which can be used as a standalone event bus in your app.
    • There's now a full test suite so people who want to hack away on Composer (including us Lyon Bros) can do so without worrying about breaking things.
    • We updated the doc site to be much better organized!

    Breaking changes

    Try as we might, we couldn't let some things stay the same and keep a clear conscience. Mainly, the problems we found were in the Router object. It no longer handles hashbang (#!) fallback...it relies completely on History.js to handle this instead. It also fixes a handful of places where non-idiomatic code was used (see below).

    • Composer.Router: the on_failure option has been removed. Instead of

      var router = new Composer.Router(routes, {on_failure: fail_fn});

      you do

      var router = new Composer.Router(routes);
router.bind('fail', fail_fn);
    • Composer.Router: The register_callback function has been removed. In order to achieve the same functionality, use router.bind('route', myfunction);.
    • Composer.Router: The "preroute" event now passes {path: path} as its argument instead of path. This allows for easier URL rewriting, but may break some apps depending on the old method.
    • Composer.Router: History.js is now a hard requirement.

    Sorry for any inconvenience this causes. However, since the rest of the framework is backwards compatible, you should be able to just use the old Composer.Router object with the new framework without any problems if you don't wish to convert your app.

    Have fun!

    Check out the new Composer.js, and please open an issue if you run into any problems. Thanks!

    - The Lyon Bros.

    Comments

  • 201407.16

    Nanomsg as the messaging layer for turtl-core

    I recently embarked on a project to rebuild the main functionality of Turtl in common lisp. This requires embedding lisp (using ECL) into node-webkit (or soon, Firefox, as node-webkit is probably getting dumped).

    To allow lisp and javascript to communicate, I made a simple messaging layer in C that both sides could easily hook into. While this worked, I stumbled on nanomsg and figured it couldn't hurt to give it a shot.

    So I wrote up some quick bindings for nanomsg in lisp and wired everything up. So far, it works really well. I can't tell if it's faster than my previous messaging layer, but one really nice thing about it is that it uses file descriptors which can be easily monitored by an event loop (such as cl-async), making polling and strange thread < --> thread event loop locking schemes a thing of the past (although cl-async handles all this fairly well).

    This simplified a lot of the Turtl code, and although right now it's only using the nanomsg "pair" layout type, it could be easily expanded in the future to allows different pieces of the app to communicate. In other words, it's a lot more future-proof than the old messaging system and probably a lot more resilient (dedicated messaging library authored by 0MQ mastermind beats hand-rolled, hard-coded simple messaging built by non-C expert).

    Comments

  • 201406.19

    Windows GUI apps: Bad file descriptor. (or how to convert a GUI app into a console app for easy debugging)

    Lately I've been neck-deep in embedding. Currently, I'm building a portable (hopefully) version of Turtl's core features in ECL.

    Problem is, when embedding turtl-core into Node-webkit or Firefox, any output that ECL writes to STDOUT triggers:

    C operation (write) signaled an error. C library explanation: Bad file descriptor.

    Well it turns out Windows doesn't let you write to STDOUT unless a console is available, and even if using msys, it doesn't create a console for GUI apps. So here's a tool (in lisp, of course) that will let you convert an executable between GUI and console.

    Seems to work great. Special thanks to death.

    Comments

  • 201402.03

    Access your Firefox extension/add-on variables from the browser console

    It can be nice to access your FF extension's variables/functions from the browser console (ctrl+shift+j) if you need some insight into its state.

    It took me a while to figure this out, so I'm sharing it. Somewhere in your extension, do:

    var chromewin = win_util.getMostRecentBrowserWindow();
chromewin.my_extension_state = ...;

    Now in the browser console, you can access whatever variables you set in the global variable my_extension_state. In my case, I used it to assign a function that lets me evaluate code in the addon's background page. This lets me gain insight into the background page's variables and state straight from the browser console.

    Note! This is a security hole. Only enable this when debugging your extension/addon. Disable it when you release it.

    Comments

  • 201401.22

    Debugging javascript in the default Android browser

    Note that this may or may not work on your device. If you're running into an app that works in a real browser but on in your Android's stock browser, do this:

    1. Navigate to your app in the browser.
    2. In the same tab go to about:debug
    3. Reload (it may reload for you).
    4. Profit.

    This will show you errors that even window.onerror doesn't catch, which should help you narrow down your problem(s).

    Source: This stackoverflow answer.

    Comments

Newer  |