The Emergence of Synaesthesia in a Neuronal Network Model via Changes in Perceptual Sensitivity and Plasticity by Oren Shriki , Yaniv Sadeh, Jamie Ward

Architecture of a general network model for studying synaesthesia.

The network consists of two interacting modalities. Each modality has a set of input neurons and a set of output neurons. There are feedforward connections within each modality but not between them. There are recurrent connections among all output neurons. The subset of recurrent connections that connect the two modalities are referred to as cross-talk interactions. The goal of the network is to optimize the representation of the combined input from both sets of input neurons, by the neurons at the combined output layer. 

For the last twenty years, theories of synaesthesia have been dominated by two general models: disinhibited feedback from multi-sensory regions to uni-sensory regions, and cross-talk theories which have emphasised the presence of atypical (and direct) structural connectivity between modalities [33]. Whereas the former explanation has tended to be favoured for explaining acquired synaesthesia, the latter has dominated explanations of developmental synaesthesia. The approach taken in our computational model represents a significant departure from this current status quo, and has generated novel insights. Our model repositions synaesthesia not as some quirk of aberrant connectivity but rather as a functional brain state that emerges, under certain conditions, as a consequence of optimising sensory information processing. In short, this model goes beyond others by offering an account not only of howsynaesthesia emerges but also of why synaesthesia emerges. It offers a unifying account of acquired and developmental forms of synaesthesia insofar as it explains how the same outcome can emerge under different conditions within the same model. 

Acquired synaesthesia is often associated with sensory deprivation due to damage to the sensory organs or pathways. Our model proposes that the same learning process that optimizes information representation naturally causes neurons in the deprived modality to enhance incoming inputs from intact modalities, leading to synaesthesia. To provide some intuition, we note that our model maximizes the output entropy of the network, which depends on two factors: one is the entropy of each single neuron, i.e. how variable the activity of single neurons is, and the other is the correlations among the neurons. Maximizing this entropy favours high single neuron entropy and low correlations among the neurons. The cross-talk connections induce correlations between the two modalities, which in general tend to reduce the output entropy. However, when one modality is deprived of input, it may be beneficial to have cross-talk connections from the intact modality to the deprived modality. The increase in the single neuron entropy due to the cross-talk connections can compensate for the higher correlations and result in a total increase of the output entropy. Loosely speaking, the deprived neurons seek for other neuronal sources of variability and enhance their connections with them. This mechanism, which emerges naturally in our computational framework, can also be useful for modelling the changes in neural representation that take place in other conditions such as phantom-limb [34]. 

Although functional accounts for acquired synaesthesia have been proposed in the past, no such comparable account has been put forward for developmental synaesthesia. Our model suggests that it arises from instability in the learning process due to high plasticity. It implies that synaesthetes have higher plasticity compared to non-synaesthetes or a relatively prolonged period of high-plasticity during childhood. Later on, as plasticity in the relevant brain areas decreases, the evolved cross-talk connections become stable. In line with this idea, whole-genome studies link some forms of synaesthesia to genes involved in plasticity, which have higher expression during early childhood [35]. Furthermore, developmental synaesthesia does not appear to be linked to sensory impairments and, if anything, is linked to increased perceptual sensitivities (notably within the concurrent modality). For instance, grapheme-colour synaesthetes show enhanced colour discrimination abilities [36]. In the proposed model, the recurrent connections within the concurrent modality amplify both its direct inputs and the ones from the inducer modality. Thus, an association between synaesthesia and increased perceptual sensitivity is an emergent property of the model, at least under certain scenarios, and it is important to explore the extent to which the presence of synaesthesia (cross-modal sensitivity) necessarily goes hand-in-hand with changes in intra-modal sensitivity. In terms of the underlying neurobiological mechanisms, the increased amplification by the recurrent interactions in our model is consistent with findings that indicate increased excitability and elevated glutamate concentration in the relevant cortical areas in synaesthetes [37, 38]. 

Traditionally, synaesthesia has not been linked to theories of learning and memory because it has been considered to reflect an innate (in its developmental form) cross-wiring of the senses. This view has been challenged on several fronts [e.g. 39, 40]. Firstly, many of the stimuli that induce synaesthesia (e.g. graphemes) are themselves learned. Secondly, for some synaesthetes the particular associations have been influenced by childhood coloured letter sets [13]. Moreover, some general cross-modal correspondences (e.g. between pitch and vertical positions) thought to reflect innate vestiges of synaesthesia have been shown to occur as statistical regularities in the environment [41]. Finally, synaesthetes (at least for grapheme-colour synaesthesia) are known to have better acquisition of new memories, and this may be related to increased plasticity during learning [32]. Future simulations of the model could use partially correlated inputs to the two modalities to model childhood exposure to coloured letter sets (they are not fully correlated given that most literacy exposure is with achromatic letters). It may well be the case that there is an interaction between learning rate (an innate parameter within the synaesthete brain) and these partial associations (in the environment), which explains why most people do not go on to develop synaesthesia after exposure to these stimuli. 

An interesting hypothesis that emerges from this study regards the relationship between synaesthesia and the concept of critical brain dynamics [28, 42, 43]. The goal of the learning process in our model is to find the pattern of recurrent interactions that maximizes the sensitivity of the network to changes in its external inputs. In analogy to physical systems, in which the sensitivity (often termed susceptibility) to external inputs diverges near a critical point [44], here, as the network maximizes its sensitivity, it also tends to approach a critical point [28]. This critical point represents the border between normal amplification of external inputs and a regime governed by attractor dynamics. In the context of sensory processing, the super-critical attractor phase can be thought of as hallucinations that reflect the learned pattern of interactions. A useful measure for identifying critical dynamics is the time it takes the recurrent network to reach steady-state. When close to critical points, many dynamical systems display the phenomenon of critical slowing down [28, 45]. Interestingly, in simulations of the complex model in which synaesthesia evolved, when the learning process approached the optimal pattern of interactions, the dynamics of the recurrent network became substantially slower (the number of iterations required to process each input sample until reaching steady-state was ~35000–45000 compared to ~1000–4000 in the beginning of the learning process). This observation suggests that in the proposed model networks that developed synaesthesia operate closer to a critical point compared to networks that did not develop synaesthesia. The prediction is that there may be evidence of the neural signatures of critical dynamics in synaesthetes [46, 47], particularly as synaesthesia is developing. 

In terms of its similarities to other models, our model resembles the direct cross-talk (or cross-activation) models proposed by others [48] primarily to account for developmental forms of synaesthesia. Although the model represents a direct form of cross-talk, it is an open question as to whether the model would produce similar patterns if neurons from modalities 1 and 2 were not directly connected but were themselves both connected via a third pool of neurons that receives no direct input from 1 and 2. There is some evidence for both direct and indirect types of neural architecture in synaesthesia as assessed via fMRI effective connectivity [49]. The addition of an interconnecting hub area in future modelling attempts would give the model top-down representations that could be adapted to the (Bayesian) predictive coding framework. Unlike the present (bottom-up) model, the predictive coding approach describes perception as top-down inference that is constrained and altered by sensory signals. A non-computationally explicit account of synaesthesia in terms of predictive coding has been articulated [50]. Moreover, the kinds of learning algorithms employed in our model are compatible with this approach [51]. 

The gradient-based learning rules used in this study are not local and are thus expected to reflect the long-term evolution of the system rather than mimicking the moment-by-moment dynamics of real neural circuits. In addition, the neurons in the model are described by simplified rate dynamics which do not capture the complex dynamics of real neurons. An important direction for future modelling work would be the examination of more biologically realistic networks that also optimize information representation. The scenarios for the evolution of synaesthesia described in this study are very general and we believe that similar scenarios would appear also in more realistic networks.

In summary, these computational models permit new ways of thinking about synaesthesia both in terms of causal mechanisms and in terms of optimising perceptual function. It generates non-trivial outcomes (e.g. generating monotonic mappings not found in the input characteristics) and non-trivial predictions (e.g. relating to learning, unimodal perceptual sensitivity, hallucinatory tendencies).

Article link

The Future of Browser History by Patryk Adaś

I am really unsatisfied with the current state of Browser History. I think that this is the most underestimated feature of every modern web browser. Let’s take the most popular browser as an example. 

Before we talk about browsers history, we need to understand how we tend to browse nowadays. 

Sometimes I want to know how to convert 1 foot to centimeters.

Sometimes, on the other hand, I want to know why and how things happen. 

For the second kind of search, the standard pattern of match retrieval does not really cut it.

Problem

I can search for the term in Google, but I’m not going to get a single result that answers my question. Rather, I’m going to get a lot of results, and all of those results will have bits and pieces of information that are relevant to me.

Then I’m going to go exploring through the internet, collecting lots of tabs along the way. Some of those tabs will be duds, so I close them.

Some of those tabs will be relevant and will have twenty more links, so I open them all, and in this way I keep crawling.

Tabs, tabs, tabs

Then after a while I have a cloud of pages in my head that I visited and the answer is more or less complete. 

But if I try to revisit this later, it’s impossible. I can remember what I found, but it wasn’t a linear progression, therefore my browser history is useless. 

Despite living in a data-driven society, as more and more databases are brought online, the complex and varied information available to be discovered is dependent on how well we can search. 

In formal ways, we have transitioned from the Classic Retrieval Model, to what is called, Berrypicking Search.

Classic Retrieval Model

The query is satisfied not by a single final retrieved set, but by a series of selections of individual references and bits of information at each stage of the ever-modifying search. 

In other words, we do not usually search for something that leads to a single result that answers our question, rather we search for terms and then explore the internet, connecting bits and pieces of the answer as we read through the web of tabs that our search starts for us.

Berrypicking Search

Our search needs, and in turn our browser history, are not being met with single query anymore. We move through a variety of sources with every new piece of information giving us new ideas and directions to follow. Without us ever knowing it, our search queries are constantly fluctuating. 

 

Current Google Chrome History

Unfortunately, our current solution to finding a not-bookmarked webpage, is to retrace own steps through different links. 

It demands that users have enough information to decipher the desired page from all others by recognizing headers, obscure URLs or timestamps. 

Our browser’s history should reflect our behaviour in the internet and help us understand the process behind it. It is crucial to actually understand and question the way we use the Internet, and without the suitable tools, it is not possible.

Solution

I find answer in maps.

Main Page

:hover

On top there is a timeline, positions are still displayed chronologically, but users can also see connections. 

Not only is this a different approach to browsing our own content, it is now possible to see patterns of my search queries and behaviors. In this way, our browser history does not only perform a retrieval function, but also writes a narrative. 

I will finally understand why I ended up reading about the influence of plants on soil properties, when I started with texas bbq query. 

With this method, I am able to see at a glance how different pieces of information are connected, how they relate to each other and how I formed conclusions. I see how I actually cognized things that are relevant. It is not only about the goal, but also about the journey. 

In the proposed interface, with a simple hover action I receive essential pieces of information. I am able to understand my thought process and points of interest. It is also easier to actually remember the page thanks to particular color schemes and meta sections. 

 

Single page 

Let’s say I already found what I was looking for and I’m satisfied with my information retrieval process. 

How many times have I visited this website? How much time have I spent using it? When did I see something interesting? How often do I visit it? How does my generated traffic look like, from perspective of particular page? 

 

Delete dropdown

Search 

Google is one of the best search engines available and yet, in Browser History there is not even a place for autosuggestion. How come? 

I’d love to search by topic, dates, colors.

Searches

In order to re-create our experience or to just see the overall topics of our interests I think it is highly useful, to provide users with a list of past searches. Once you click a particular search result, it expands with visited links based on that very query and would redirect you to the map with the highlighted path.

Analytics 

Currently, I am really missing the analytics screen. It is crucial to be able to understand one’s own behavior, especially as there is no distinction between offline and online anymore. Filter Bubble shows that information we see is selective, we become separated from information that disagrees with our viewpoint. Users are more and more comforted in their own domain, we have stopped questioning. 

I want to see how much time I spend browsing Internet, how I collect information and how I form my views. 

We live in times when understanding your browse behaviors and search patterns is becoming crucial in cognitivism process. 

I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail. 

By Patryk Adaś

The UX-factor: why I don’t care where or what you studied by Bram Bos

Trolling design establishment. Like. A. Boss.

I vividly remember the first time I was about to build my own UX design team for a big multinational company. When the hiring process started, the lady from recruitment asked me what sort of educational background I was looking for in potential candidates. My answer: “I don’t particularly care”.

My senior creative director, a well-respected authority in product design (and in many ways a mentor to me) with many decades of experience all but exploded with rage. He prided himself in always hiring the most promising talents from the most prestigious design institutes. Of course it mattered what background a candidate has!

Since then I’ve done my share of hiring UX/UI design talent, and I still stand by my statement: I don’t care what background you have. Just show me you have the UX-factor.

We’re building this brave new world together

Back when I started as an interaction designer in the early 90s, the field of expertise barely existed. As I wrote in another article, we were rebels; using design to avenge the atrocities of technocracy from the 80s. The eternally blinking 00:00 on VCRs, excessively button-laden copy machines, the “underwater” screens of Wordperfect, the insufferable arrogance of the DOS prompt.

In the 20 years that followed, interaction design evolved into user experience design and a smorgasbord of related competencies. The most influential contributors of the field came from myriad backgrounds: human factors, graphic design, architecture, AV arts, software engineering, philosophy, cognitive psychology, people research. I can’t think of any other profession that encompasses such a holistic spectrum of arts and sciences.

But since the founding fathers and mothers of the UX expertise are so diverse, why should we narrow the process of UX design recruitment down to people with a highly specific background? Unless I’m looking to fill a very specific vacancy (e.g. UX trend research, or user testing) I’m just as open to interviewing candidates with a background in architecture or scenography as, say, someone who was trained as a graphic designer but taught herself interaction design skills.

So what is the UX-factor?

Educational background is obviously not completely irrelevant. It will show me things about your interests, the context in which your skills were initially formed, whether you’re likely taught to take an academic approach versus an intuitive approach, etc. But when it comes down to it, for me it’s all about the portfolio. Show me what you’ve done. Tell me about the process and your ideas. And while we’re talking, I’m hoping to spot three things - the trinity of a true UX designer’s character traits:

1. A passion for design
2. Empathy for people
3. A knack for problem solving

Passion for design

While interaction design (with its roots in human factors and HCI) has always been a pretty rational specialism, User Experience design -by its very definition- is an expertise that makes technology and communication resonate with people on a rational AND on an emotional level. It has to.

You can design something to be 100% usable and logical to your end-user, but if the product is dull-as-dishwater people may still have no interest in using it. Don’t underestimate the irrational side of people; the importance of warm fuzzy feelings. (And that’s why I feel user testing is a double-edged sword; it often ignores the unquantifiable qualities of design. But that’s a discussion for another article).

Don’t underestimate the irrational side of people; the importance of warm fuzzy feelings.

I believe that the passion a designer puts into her work always shines through in the end-result. The fun you have making something will be reflected somehow, no matter how mundane or utilitarian the product you’re working on. That’s why it’s essential for a designer to truly have a passion for the irrational, the beautiful, the elegant, the superficial, the emotional, the unnecessary, the enjoyable. It’s what will make technology feel human.

Empathy for people

As a user experience designer you have to identify with the people you design for. Empathy is much more than just understanding your user. If you understand your user, you’re half-way there. It means you know what people do and the logical rationale behind those actions — a workflow, if you like.

But you still don’t really know why people do things, what drives them emotionally. Is it enjoyment, fear, boredom, pride, convenience or curiosity? Knowing such motivations can help your design tremendously. It can make the difference between indifference and engagement. For example: it’s what makes the difference between a wearable step-counter and a wearable that motivates people to sustainably change their behavior.

But you still don’t really know why people do things, what drives them emotionally. Is it enjoyment, fear, boredom, pride, convenience or curiosity?

Feeling a bond with your user is an absolute must to take your designs beyond their rational purpose.

A knack for problem solving

I recently read an article claiming the definition of design is “solving problems”. I think that’s both an oversimplification and a disservice to the wonderful kaleidoscope of facets that make up design. Having said that, creative problem solving is certainly one of the defining qualities of design. And especially in UX design and interaction design there are often many problems to solve and obstacles to remove.

The Tao of user experience design is a concept called “flow”. In a perfect user experience all tasks flow seamlessly and uninterruptedly into each other. All obstructions have been removed, all friction is smoothed out by the designer, there are no perceivable hoops to jump through.

The Tao of user experience design is a concept called “flow”. In a perfect user experience all tasks flow seamlessly and uninterruptedly into each other.

This means problems are not just solved, but the solutions are seamlessly embedded into a bigger context. It takes a designer that can zoom in and out of problems: draw the blueprint for a complex construct of tasks, but also come up with clever micro-interactions.

Serving the Trinity

As I stated earlier, we’re building this brave new world together. I don’t believe the field of User Experience design has finished evolving. We’re not even close. This means there is still a lot of opportunity for cross-pollination with other professions, arts and sciences. And none of the three characteristics I described are exclusive to a single design competence. They can exist in many different professional contexts. So, I don’t care what your background is, as long as you love design, care about people and surprise me with clever ideas.

---

Bram Bos

Creative Director User Experience, World Traveler, Perpetual Learner, Amateur Tinkerer, Occasional Writer