.. _similarity:

###################
Similarity analysis
###################

.. testsetup::

   >>> dmm_path = str(outdir / "dmm.svg")
   >>> smdm1_path = str(outdir / "smdm1.svg")
   >>> smdm2_path = str(outdir / "smdm2.svg")
   >>> smdm3_path = str(outdir / "smdm3.svg")


The design of a product family requires the creation of a basic product architecture
from which all product family members can be derived. By analyzing the similarity of
products within an existing product portfolio one can get an indication of which and how
many product variants have to be supported by the architecture.


*****
Input
*****

The input for a similarity analysis is a domain mapping matrix (or bipartite graph) that
maps individual products from the portfolio to attributes that characterize the
functionality and designs of the products. The :obj:`~ragraph.datasets` module contains
an example data set for performing a similarity analysis. So let's get started and load
the 'similarity' dataset:

.. doctest::

   >>> from ragraph import datasets
   >>> graph = datasets.get("similarity")


and subsequently visualize it using :obj:`~ragraph.plot.dmm`:

.. doctest::

   >>> import ragraph.plot
   >>> fig = ragraph.plot.dmm(
   ...     rows=[n for n in graph.nodes if n.kind == "attribute"],
   ...     cols=[n for n in graph.nodes if n.kind == "product"],
   ...     edges=graph.edges,
   ...     sort=False,
   ... )
   >>> fig.write_image(dmm_path)  # e.g.: "dmm.svg"


which results in :numref:`PA-DMM`, which shows the resulting domain mapping matrix
(DMM), in which twelve products (columns) are mapped to ten attributes (rows). A mark at
position :math:`i,j` indicates that product :math:`j` is characterized by attribute
:math:`i`.

.. figure:: /_static/generated/dmm.svg
   :alt: Product - attribute domain mapping matrix.
   :name: PA-DMM
   :align: center

   Product - attribute domain mapping matrix.


In selecting attributes for performing a similarity analysis you should balance the
attribute granularity level. That is, very fine grained (detailed) attributes will yield
a very sparse mapping matrix, while very coarse (high level) attributes will yield a
very dense mapping matrix. Moreover, you should take care in ensuring that the
attributes are non-overlapping. In general, it is advised to use attributes that define
functionality, working-principles, and embodiments of product(s) (modules).

Once you have defined your graph, mapping products to attributes, you can use the
:obj:`~ragraph.analysis.similarity.SimilarityAnalysis` object to perform the similarity
analysis. First, you have to instantiate the
:obj:`~ragraph.analysis.similarity.SimilarityAnalysis` object for which a minimal
example is shown below:

.. doctest::

   >>> from ragraph.analysis.similarity import SimilarityAnalysis
   >>> sa = SimilarityAnalysis(
   ...     rows=[n for n in graph.nodes if n.kind == "attribute"],
   ...     cols=[n for n in graph.nodes if n.kind == "product"],
   ...     edges=graph.edges,
   ... )


The :obj:`~ragraph.analysis.similarity.SimilarityAnalysis` object requires three
parameters: :obj:`rows` which is a list of :obj:`~ragraph.node.Node` objects which are
the row elements of a DMM (in this case attributes), :obj:`cols` which is a list of
:obj:`~ragraph.node.Node` objects which are the column elements of a DMM (in this case
products),  and :obj:`edges` which is a list of :obj:`~ragraph.edge.Edge` objects that
map column elements to row elements.

Internally, the :obj:`~ragraph.analysis.similarity.jaccard_matrix` is calculated for
both the columns elements as the row elements, which are stored within the
:obj:`~ragraph.analysis.similarity.SimilarityAnalysis.row_similarity_matrix` and
:obj:`~ragraph.analysis.similarity.SimilarityAnalysis.col_similarity_matrix` attributes:

.. doctest::

   >>> print(sa.row_similarity_matrix)
   [[1.   0.   0.   0.25 0.   0.25 0.17 0.2  0.   0.  ]
    [0.   1.   0.   0.   0.   0.   0.2  0.   0.   0.33]
    [0.   0.   1.   0.   0.5  0.   0.17 0.   0.25 0.  ]
    [0.25 0.   0.   1.   0.   0.   0.   0.25 0.   0.  ]
    [0.   0.   0.5  0.   1.   0.   0.17 0.   0.25 0.  ]
    [0.25 0.   0.   0.   0.   1.   0.2  0.25 0.   0.  ]
    [0.17 0.2  0.17 0.   0.17 0.2  1.   0.   0.   0.2 ]
    [0.2  0.   0.   0.25 0.   0.25 0.   1.   0.   0.  ]
    [0.   0.   0.25 0.   0.25 0.   0.   0.   1.   0.  ]
    [0.   0.33 0.   0.   0.   0.   0.2  0.   0.   1.  ]]
   >>> print(sa.col_similarity_matrix)
   [[1.   0.25 0.33 0.   0.33 0.   0.   0.   0.   0.25 0.   0.  ]
    [0.25 1.   0.   0.25 0.25 0.25 0.25 0.   0.   0.2  0.   0.  ]
    [0.33 0.   1.   0.   0.33 0.   0.   0.   0.   0.   0.   0.  ]
    [0.   0.25 0.   1.   0.   0.   0.33 0.   0.   0.   0.33 0.  ]
    [0.33 0.25 0.33 0.   1.   0.   0.   0.   0.   0.25 0.   0.  ]
    [0.   0.25 0.   0.   0.   1.   0.33 0.   0.   0.   0.33 0.  ]
    [0.   0.25 0.   0.33 0.   0.33 1.   0.   0.   0.   0.33 0.  ]
    [0.   0.   0.   0.   0.   0.   0.   1.   0.33 0.25 0.   0.33]
    [0.   0.   0.   0.   0.   0.   0.   0.33 1.   0.67 0.   0.33]
    [0.25 0.2  0.   0.   0.25 0.   0.   0.25 0.67 1.   0.   0.25]
    [0.   0.   0.   0.33 0.   0.33 0.33 0.   0.   0.   1.   0.  ]
    [0.   0.   0.   0.   0.   0.   0.   0.33 0.33 0.25 0.   1.  ]]


The data from these matrices are stored within the internal
:obj:`~ragraph.analysis.similarity.SimilarityAnalysis.graph` property. This graph is a
regular :obj:`~ragraph.graph.Graph` object which you can be visualized using the
:obj:`~ragraph.plot.mdm` module:

.. doctest::

   >>> import ragraph.plot
   >>> fig = ragraph.plot.mdm(
   ...     leafs=sa.graph.leafs,
   ...     edges=sa.graph.edges,
   ...     style=ragraph.plot.Style(
   ...         piemap=dict(
   ...             display="weights",
   ...             fields=["similarity"]
   ...         )
   ...     ),
   ... )
   >>> fig.write_image(smdm1_path)


:numref:`SMDM1` shows the resulting product - attribute multi-domain-matrix (MDM) in
which similarity weights are displayed.

.. figure:: /_static/generated/smdm1.svg
   :alt: Product - attribute similarity multi-domain-matrix.
   :name: SMDM1
   :align: center

   Product - attribute multi-domain-matrix showing similarity weights.


*******
Pruning
*******

If a similarity matrix is very dense, you may want to prune the matrix by removing all
values below a certain threshold. You can prune the matrices by setting the values of
the :obj:`~ragraph.analysis.similarity.SimilarityAnalysis.col_sim_threshold` and
:obj:`~ragraph.analysis.similarity.SimilarityAnalysis.row_sim_threshold` attributes:

.. doctest::

   >>> sa.col_sim_threshold = 0.30
   >>> sa.row_sim_threshold = 0.20


By changing the value of these attributes, the
:obj:`~ragraph.analysis.similarity.SimilarityAnalysis.graph` attribute will
automatically update, as show in :numref:`SMDM2` in which all edges with a similarity
weight below the respective threshold are removed.

.. doctest::

   >>> import ragraph.plot
   >>> fig = ragraph.plot.mdm(
   ...     leafs=sa.graph.leafs,
   ...     edges=sa.graph.edges,
   ...     style=ragraph.plot.Style(
   ...         piemap=dict(
   ...             display="weights",
   ...             fields=["similarity"]
   ...         )
   ...     ),
   ... )
   >>> fig.write_image(smdm2_path)


.. figure:: /_static/generated/smdm2.svg
   :alt: Pruned product - attribute similarity multi-domain-matrix.
   :name: SMDM2
   :align: center

   Pruned product - attribute multi-domain-matrix showing similarity weights.


**********
Clustering
**********

The aim of a similarity analysis is to find groups (clusters) of products that are
similar and could therefore be standardized. Similarly, one could search for groups
(clusters) of attributes that have high similarity, which implies that they are often
found together within products. To highlight these clusters you can use the
:obj:`~ragraph.analysis.similarity.SimilarityAnalysis.cluster_rows` and
:obj:`~ragraph.analysis.similarity.SimilarityAnalysis.cluster_cols` methods:

.. doctest::

   >>> sa.cluster_rows(alpha=2.0, beta=2.0, mu=2.0)
   >>> sa.cluster_cols(alpha=2.0, beta=2.0, mu=2.0)


By default the :obj:`~ragraph.analysis.cluster.markov` algorithm. You could, however,
provide a different algorithm by setting the :obj:`algo` argument if desired.

After clustering, you can re-visualize the product - attribute MDM:

.. doctest::

   >>> import ragraph.plot
   >>> sa.row_sim_threshold = 0.0
   >>> sa.col_sim_threshold = 0.0
   >>> fig = ragraph.plot.mdm(
   ...     leafs=sa.graph.leafs,
   ...     edges=sa.graph.edges,
   ...     style=ragraph.plot.Style(
   ...         piemap=dict(
   ...             display="weights",
   ...             fields=["similarity"]
   ...         )
   ...     ),
   ... )
   >>> fig.write_image(smdm3_path)  # e.g.: "dmm.svg"


Note that we reset the similarity thresholds to 0.0 to ensure that all data is
displayed. :numref:`SMDM3` shows the resulting MDM in which one can observe three
product clusters and three attribute clusters. The first product cluster maps to the
first attribute cluster, the second product cluster primarily maps to the third
attribute cluster, and the third product cluster primarily maps to the second attribute
clusters. This indicates the presences of three product families within the product
portfolio, each with a distinct set of attributes that characterize their functionality
and design.

.. figure:: /_static/generated/smdm3.svg
   :alt: Clustered product - attribute similarity multi-domain-matrix.
   :name: SMDM3
   :align: center

   Clustered product - attribute similarity multi-domain-matrix.