Matrix 源码分析 ———— Apk Canary

概述

年前，微信开源了Matrix项目，提供了Android、ios的APM实现方案。对于Android端实现，主要包括APK Checker、Resource Canary、Trace Canary、SQLite Lint、IO Canary五部分。本文主要介绍APK Checker的源码实现，其他部分的源码分析将在后续推出。

代码框架分析

整体代码结构比较清晰，主要包括三部分：ApkJob、Task、 Result。ApkJob是表示整体这个apk检测任务，Task表示每一步细分的检测任务、Result表示检测任务的结果。总体流程如下：ApkJob读取配置信息，实例化相关的Task任务；相关Task任务执行之后输出Result到文件（默认为MMTaskJsonResult）。

Task任务实现分析

UnZipTask

目的：解压Apk，解析Class混淆规则、Res混淆规则，并输出apk中每个entry原始大小、zip包中压缩后的大小。主要存储了一些原始数据，为后续的Task做准备。

@Override
    public TaskResult call() throws TaskExecuteException {

        try {
            //apk文件
            ZipFile zipFile = new ZipFile(inputFile);
            ...
            //Result输出对象
            TaskResult taskResult = TaskResultFactory.factory(getType(), TASK_RESULT_TYPE_JSON, config);
            ...
            //apk总大小
            ((TaskJsonResult) taskResult).add("total-size", inputFile.length());
            //读取Class的mapping规则，并存储到config对象中
            readMappingTxtFile();
            config.setProguardClassMap(proguardClassMap);
            //读取Res的mapping规则，并存储到config对象中
            readResMappingTxtFile();
            config.setResguardMap(resguardMap);

            Enumeration entries = zipFile.entries();
            JsonArray jsonArray = new JsonArray();
            String outEntryName = "";
            while (entries.hasMoreElements()) {
                ZipEntry entry = (ZipEntry) entries.nextElement();
                outEntryName = writeEntry(zipFile, entry);
                if (!Util.isNullOrNil(outEntryName)) {
                    JsonObject fileItem = new JsonObject();
                    //输出Apk中每个item的名字、压缩后的大小
                    fileItem.addProperty("entry-name", outEntryName);
                    fileItem.addProperty("entry-size", entry.getCompressedSize());
                    jsonArray.add(fileItem);
                    //Map：解压后文件（相对路径）-> (未压缩Size，压缩后Size)
                    entrySizeMap.put(outEntryName, Pair.of(entry.getSize(), entry.getCompressedSize()));
                    //Map：Apk中文件名 -> ：解压后文件（相对路径）
                    entryNameMap.put(entry.getName(), outEntryName);
                }
            }
            //存储到config对象
            config.setEntrySizeMap(entrySizeMap);
            config.setEntryNameMap(entryNameMap);
            //输出到Result
            ((TaskJsonResult) taskResult).add("entries", jsonArray);
            taskResult.setStartTime(startTime);
            taskResult.setEndTime(System.currentTimeMillis());
            return taskResult;
        } catch (Exception e) {
            throw new TaskExecuteException(e.getMessage(), e);
        }
    }

重点讲解一下Task任务中mapping文件的解析规则：

• Class Mapping

class mapping文件截取片段：

    ...
android.arch.core.executor.ArchTaskExecutor$1 -> android.arch.a.a.a$1:
    42:42:void <init>() -> <init>
    45:46:void execute(java.lang.Runnable) -> execute
android.arch.core.executor.ArchTaskExecutor$2 -> android.arch.a.a.a$2:
    50:50:void <init>() -> <init>
    53:54:void execute(java.lang.Runnable) -> execute
android.arch.core.executor.DefaultTaskExecutor -> android.arch.a.a.b:
    java.lang.Object mLock -> a
    java.util.concurrent.ExecutorService mDiskIO -> b
    android.os.Handler mMainHandler -> c
    31:33:void <init>() -> <init>
    40:41:void executeOnDiskIO(java.lang.Runnable) -> a
    45:54:void postToMainThread(java.lang.Runnable) -> b
    58:58:boolean isMainThread() -> b
    ...

* 原始类名 -> 混淆后类名 （顶格）
* 原始字段名 -> 混淆后字段名   （行首预留一个Tab）
* 原始函数名 -> 混淆后函数名   （行首预留一个Tab）

• Res Mapping

res mapping文件截取片段：

res path mapping:
    res/layout-v22 -> r/a
    res/drawable -> r/b
    res/color-night-v8 -> r/c
    res/xml -> r/d
    res/layout -> r/e
  ...
  
res id mapping:
    com.example.app.R.attr.avatar_border_color -> com.example.app.R.attr.a
    com.example.app.R.attr.actualImageScaleType -> com.example.app.R.attr.b
    com.example.app.R.attr.backgroundImage -> com.example.app.R.attr.c
    com.example.app.R.attr.fadeDuration -> com.example.app.R.attr.d
    com.example.app.R.attr.failureImage -> com.example.app.R.attr.e

* 原始资源目录 -> 混淆后资源目录
* 原始资源名 -> 混淆后资源名

ManifestAnalyzeTask

目的：解析Manifest文件、arsc文件

public TaskResult call() throws TaskExecuteException {
        try {
            ManifestParser manifestParser = null;
            //创建Manifest解析对象
            if (!FileUtil.isLegalFile(arscFile)) {
                manifestParser = new ManifestParser(inputFile);
            } else {
                manifestParser = new ManifestParser(inputFile, arscFile);
            }
            TaskResult taskResult = TaskResultFactory.factory(getType(), TASK_RESULT_TYPE_JSON, config);
            if (taskResult == null) {
                return null;
            }
            long startTime = System.currentTimeMillis();
            JsonObject jsonObject = manifestParser.parse();
            //输出Manifest解析结果
            ((TaskJsonResult) taskResult).add("manifest", jsonObject);
            taskResult.setStartTime(startTime);
            taskResult.setEndTime(System.currentTimeMillis());
            return taskResult;
        } catch (Exception e) {
            throw new TaskExecuteException(e.getMessage(), e);
        }
    }

此处讲解下arsc文件。arsc文件以二进制形式存在，存储了资源的索引信息，基本文件格式如下图（图片来源网络）：

用二进制工具查看arsc文件的内容：

arsc的详细文件格式暂时不展开，可参考文章，此处仅简单分析一下二进制工具中可视化展示的一些信息。

• ResTable_header部分

• ResStringPool部分

• ResTablePackage

关于arsc文件解析的相关内容，详见文章

ShowFileSizeTask

目的：统计超过阈值的文件。

public TaskResult call() throws TaskExecuteException {
        ...
            long startTime = System.currentTimeMillis();
            //获取UnZipTask中记录的 文件名->（文件压缩后大小，文件压缩前大小） map
            Map<String, Pair<Long, Long>> entrySizeMap = config.getEntrySizeMap();
            if (!entrySizeMap.isEmpty()) {                                        
                for (Map.Entry<String, Pair<Long, Long>> entry : entrySizeMap.entrySet()) {
                    final String suffix = getSuffix(entry.getKey());
                    Pair<Long, Long> size = entry.getValue();
                    // 记录超出阈值的文件
                    if (size.getFirst() >= downLimit * ApkConstants.K1024) {
                        if (filterSuffix.isEmpty() || filterSuffix.contains(suffix)) {
                            entryList.add(Pair.of(entry.getKey(), size.getFirst()));
                        } 
                    } 
                }
            }

           ...
           //排序

            JsonArray jsonArray = new JsonArray();
            for (Pair<String, Long> sortFile : entryList) {
                JsonObject fileItem = new JsonObject();
                fileItem.addProperty("entry-name", sortFile.getFirst());
                fileItem.addProperty("entry-size", sortFile.getSecond());
                jsonArray.add(fileItem);
            }
            //输出到结果
            ((TaskJsonResult) taskResult).add("files", jsonArray);
            taskResult.setStartTime(startTime);
            taskResult.setEndTime(System.currentTimeMillis());
            return taskResult;
        } catch (Exception e) {
            throw new TaskExecuteException(e.getMessage(), e);
        }
    }

MethodCountTask

目的：统计在本dex文件内定义的方法数、未在本dex文件内定义的方法数。

public TaskResult call() throws TaskExecuteException {
        try {
            ...
            long startTime = System.currentTimeMillis();
            JsonArray jsonArray = new JsonArray();
            for (int i = 0; i < dexFileList.size(); i++) {
                RandomAccessFile dexFile = dexFileList.get(i);
                //计算dex中的方法信息
                countDex(dexFile);
                //dex内能找到定义的方法
                int totalInternalMethods = sumOfValue(classInternalMethod);
                //跨dex的方法
                int totalExternalMethods = sumOfValue(classExternalMethod);
                JsonObject jsonObject = new JsonObject();
                jsonObject.addProperty("dex-file", dexFileNameList.get(i));
                //按Class维度聚合
                if (JobConstants.GROUP_CLASS.equals(group)) {
                    List<String> sortList = sortKeyByValue(classInternalMethod);
                    JsonArray classes = new JsonArray();
                    for (String className : sortList) {
                        JsonObject classObj = new JsonObject();
                        classObj.addProperty("name", className);
                        classObj.addProperty("methods", classInternalMethod.get(className));
                        classes.add(classObj);
                    }
                    jsonObject.add("internal-classes", classes);
                    //按package维度聚合
                } else if (JobConstants.GROUP_PACKAGE.equals(group)) {
                    String packageName;
                    for (Map.Entry<String, Integer> entry : classInternalMethod.entrySet()) {
                        packageName = ApkUtil.getPackageName(entry.getKey());
                        if (!Util.isNullOrNil(packageName)) {
                            if (!pkgInternalRefMethod.containsKey(packageName)) {
                                pkgInternalRefMethod.put(packageName, entry.getValue());
                            } else {
                                pkgInternalRefMethod.put(packageName, pkgInternalRefMethod.get(packageName) + entry.getValue());
                            }
                        }
                    }
                    List<String> sortList = sortKeyByValue(pkgInternalRefMethod);
                    JsonArray packages = new JsonArray();
                    for (String pkgName : sortList) {
                        JsonObject pkgObj = new JsonObject();
                        pkgObj.addProperty("name", pkgName);
                        pkgObj.addProperty("methods", pkgInternalRefMethod.get(pkgName));
                        packages.add(pkgObj);
                    }
                    jsonObject.add("internal-packages", packages);
                }
                jsonObject.addProperty("total-internal-classes", classInternalMethod.size());
                jsonObject.addProperty("total-internal-methods", totalInternalMethods);

                if (JobConstants.GROUP_CLASS.equals(group)) {
                    List<String> sortList = sortKeyByValue(classExternalMethod);
                    JsonArray classes = new JsonArray();
                    for (String className : sortList) {
                        JsonObject classObj = new JsonObject();
                        classObj.addProperty("name", className);
                        classObj.addProperty("methods", classExternalMethod.get(className));
                        classes.add(classObj);
                    }
                    jsonObject.add("external-classes", classes);

                } else if (JobConstants.GROUP_PACKAGE.equals(group)) {
                    String packageName = "";
                    for (Map.Entry<String, Integer> entry : classExternalMethod.entrySet()) {
                        packageName = ApkUtil.getPackageName(entry.getKey());
                        if (!Util.isNullOrNil(packageName)) {
                            if (!pkgExternalMethod.containsKey(packageName)) {
                                pkgExternalMethod.put(packageName, entry.getValue());
                            } else {
                                pkgExternalMethod.put(packageName, pkgExternalMethod.get(packageName) + entry.getValue());
                            }
                        }
                    }
                    List<String> sortList = sortKeyByValue(pkgExternalMethod);
                    JsonArray packages = new JsonArray();
                    for (String pkgName : sortList) {
                        JsonObject pkgObj = new JsonObject();
                        pkgObj.addProperty("name", pkgName);
                        pkgObj.addProperty("methods", pkgExternalMethod.get(pkgName));
                        packages.add(pkgObj);
                    }
                    jsonObject.add("external-packages", packages);

                }
                jsonObject.addProperty("total-external-classes", classExternalMethod.size());
                jsonObject.addProperty("total-external-methods", totalExternalMethods);
                jsonArray.add(jsonObject);
            }
            ((TaskJsonResult) taskResult).add("dex-files", jsonArray);
            taskResult.setStartTime(startTime);
            taskResult.setEndTime(System.currentTimeMillis());
            return taskResult;
        } catch (Exception e) {
            throw new TaskExecuteException(e.getMessage(), e);
        }
    }

这段代码的重点是如何对dex文件进行静态分析的

private void countDex(RandomAccessFile dexFile) throws IOException {
        classInternalMethod.clear();
        classExternalMethod.clear();
        pkgInternalRefMethod.clear();
        pkgExternalMethod.clear();
        DexData dexData = new DexData(dexFile);
        //加载dex数据
        dexData.load();
        MethodRef[] methodRefs = dexData.getMethodRefs();
        ClassRef[] externalClassRefs = dexData.getExternalReferences();
        //获取混淆的Class maping规则
        Map<String, String> proguardClassMap = config.getProguardClassMap();
        String className = null;
        for (ClassRef classRef : externalClassRefs) {
            className = ApkUtil.getNormalClassName(classRef.getName());
            if (proguardClassMap.containsKey(className)) {
                //混淆前的原始className
                className = proguardClassMap.get(className);
            }
            if (className.indexOf('.') == -1) {
                continue;
            }
            classExternalMethod.put(className, 0);
        }
        for (MethodRef methodRef : methodRefs) {
            className = ApkUtil.getNormalClassName(methodRef.getDeclClassName());
            if (proguardClassMap.containsKey(className)) {
                className = proguardClassMap.get(className);
            }
            if (!Util.isNullOrNil(className)) {
                if (className.indexOf('.') == -1) {
                    continue;
                }
                if (classExternalMethod.containsKey(className)) {
                    classExternalMethod.put(className, classExternalMethod.get(className) + 1);
                } else if (classInternalMethod.containsKey(className)) {
                    classInternalMethod.put(className, classInternalMethod.get(className) + 1);
                } else {
                    classInternalMethod.put(className, 1);
                }
            }
        }

        //remove 0-method referenced class
        Iterator<String> iterator = classExternalMethod.keySet().iterator();
        while (iterator.hasNext()) {
            if (classExternalMethod.get(iterator.next()) == 0) {
                iterator.remove();
            }
        }
    }

理解上述代码之前，先介绍下dex文件格式。 dex文件可分为Header部分、String索引表、类型索引表、方法原型索引表、字段索引表、方法索引表、类定义、Data数据区。

• Header部分

• String索引表

• 类型索引表

• 方法原型索引表

• 字段索引表

• 方法索引表

• 类定义

通过二进制工具，大概讲解了dex的文件格式。再回过头看代码，代码中有一个classInternalMethod和classExternalMethod的区别；首先在解析TypeId的时候会有一个internal字段表示这个类型是否定义在这个dex文件内；

  /**
     * Holds the contents of a type_id_item.
     *
     * This is chiefly a list of indices into the string table.  We need
     * some additional bits of data, such as whether or not the type ID
     * represents a class defined in this DEX, so we use an object for
     * each instead of a simple integer.  (Could use a parallel array, but
     * since this is a desktop app it's not essential.)
     */
    static class TypeIdItem {
        public int descriptorIdx;       // index into string_ids

        public boolean internal;        // defined within this DEX file?
    }

internal字段的赋值操作如下：

 /**
     * Sets the "internal" flag on type IDs which are defined in the
     * DEX file or within the VM (e.g. primitive classes and arrays).
     */
    void markInternalClasses() {
        for (int i = mClassDefs.length - 1; i >= 0; i--) {
            mTypeIds[mClassDefs[i].classIdx].internal = true;
        }

        for (int i = 0; i < mTypeIds.length; i++) {
            String className = mStrings[mTypeIds[i].descriptorIdx];

            if (className.length() == 1) {
                // primitive class
                mTypeIds[i].internal = true;
            } else if (className.charAt(0) == '[') {
                mTypeIds[i].internal = true;
            }

            //System.out.println(i + " " +
            //    (mTypeIds[i].internal ? "INTERNAL" : "external") + " - " +
            //    mStrings[mTypeIds[i].descriptorIdx]);
        }
    }

在ClassDef中定义的类型都属于internal，同时转换后的className长度为1的类型(基础数据类型)也属于interal，最后数组类型的也属于internal。

classInternalMethod和classExternalMethod的具体划分规则如下：

  private void countDex(RandomAccessFile dexFile) throws IOException {
       ...
       ...
        for (ClassRef classRef : externalClassRefs) {
            className = ApkUtil.getNormalClassName(classRef.getName());
            if (proguardClassMap.containsKey(className)) {
                className = proguardClassMap.get(className);
            }
            if (className.indexOf('.') == -1) {
                continue;
            }
            //将类定义不在本dex文件中的类名加入map
            classExternalMethod.put(className, 0);
        }
        for (MethodRef methodRef : methodRefs) {
            className = ApkUtil.getNormalClassName(methodRef.getDeclClassName());
            if (proguardClassMap.containsKey(className)) {
                className = proguardClassMap.get(className);
            }
            if (!Util.isNullOrNil(className)) {
                if (className.indexOf('.') == -1) {
                    continue;
                }
                //根据类名加入不同的分类
                if (classExternalMethod.containsKey(className)) {
                    classExternalMethod.put(className, classExternalMethod.get(className) + 1);
                } else if (classInternalMethod.containsKey(className)) {
                    classInternalMethod.put(className, classInternalMethod.get(className) + 1);
                } else {
                    classInternalMethod.put(className, 1);
                }
            }
        }

        //remove 0-method referenced class
        Iterator<String> iterator = classExternalMethod.keySet().iterator();
        while (iterator.hasNext()) {
            if (classExternalMethod.get(iterator.next()) == 0) {
                iterator.remove();
            }
        }
    }

ResProguardCheckTask

目的：判断apk是否执行了资源混淆。

 @Override
    public TaskResult call() throws TaskExecuteException {
        File resDir = new File(inputFile, ApkConstants.RESOURCE_DIR_PROGUARD_NAME);
            ...
            if (resDir.exists() && resDir.isDirectory()) {
                Log.d(TAG, "find resource directory " + resDir.getAbsolutePath());
                //有名为r的文件夹，执行了支援混淆
                ((TaskJsonResult) taskResult).add("hasResProguard", true);
            } else {
                resDir = new File(inputFile, ApkConstants.RESOURCE_DIR_NAME);
                if (resDir.exists() && resDir.isDirectory()) {
                    File[] dirs = resDir.listFiles();
                    boolean hasProguard = true;
                    for (File dir : dirs) {
                        //任意文件夹不符合资源混淆的命名规则，则未执行资源混淆
                        if (dir.isDirectory() && !fileNamePattern.matcher(dir.getName()).matches()) {
                            hasProguard = false;
                            Log.i(TAG, "directory " + dir.getName() + " has a non-proguard name!");
                            break;
                        }
                    }
                    ((TaskJsonResult) taskResult).add("hasResProguard", hasProguard);
             ...
    }

FindNonAlphaPngTask

目的：检测出没有透明度的png文件（应该使用jpg替换，占用空间会更小）

private void findNonAlphaPng(File file) throws IOException {
        if (file != null) {
            if (file.isDirectory()) {
                File[] files = file.listFiles();
                for (File tempFile : files) {
                    findNonAlphaPng(tempFile);
                }
            } else if (file.isFile() && file.getName().endsWith(ApkConstants.PNG_FILE_SUFFIX) && !file.getName().endsWith(ApkConstants.NINE_PNG)) {
                BufferedImage bufferedImage = ImageIO.read(file);
                //没有alpha信息
                if (!bufferedImage.getColorModel().hasAlpha()) {
                    String filename = file.getAbsolutePath().substring(inputFile.getAbsolutePath().length() + 1);
                    if (entryNameMap.containsKey(filename)) {
                        filename = entryNameMap.get(filename);
                    }
                    long size = file.length();
                    if (entrySizeMap.containsKey(filename)) {
                        size = entrySizeMap.get(filename).getFirst();
                    }
                    if (size >= downLimitSize * ApkConstants.K1024) {
                        nonAlphaPngList.add(Pair.of(filename, file.length()));
                    }
                }
            }
        }
    }

MultiLibCheckTask

目的：检测lib文件夹中是否有多文件夹存在。

    @Override
    public TaskResult call() throws TaskExecuteException {
        try {
            TaskResult taskResult = TaskResultFactory.factory(getType(), TASK_RESULT_TYPE_JSON, config);
            if (taskResult == null) {
                return null;
            }
            long startTime = System.currentTimeMillis();
            JsonArray jsonArray = new JsonArray();
            if (libDir.exists() && libDir.isDirectory()) {
                File[] dirs = libDir.listFiles();
                for (File dir : dirs) {
                    if (dir.isDirectory()) {
                        jsonArray.add(dir.getName());
                    }
                }
            }
            ((TaskJsonResult) taskResult).add("lib-dirs", jsonArray);
            if (jsonArray.size() > 1) {
                ((TaskJsonResult) taskResult).add("multi-lib", true);
            } else {
                ((TaskJsonResult) taskResult).add("multi-lib", false);
            }
            taskResult.setStartTime(startTime);
            taskResult.setEndTime(System.currentTimeMillis());
            return taskResult;
        } catch (Exception e) {
            throw new TaskExecuteException(e.getMessage(), e);
        }
    }

UncompressedFileTask

目的：比对apk压缩包里每一个entry的压缩后大小、压缩前大小；若大小一样，则表示文件未压缩。

@Override
    public TaskResult call() throws TaskExecuteException {
        try {
            ...
            if (!entrySizeMap.isEmpty()) {                                                          //take advantage of the result of UnzipTask.
                for (Map.Entry<String, Pair<Long, Long>> entry : entrySizeMap.entrySet()) {
                    final String suffix = getSuffix(entry.getKey());
                    Pair<Long, Long> size = entry.getValue();
                    if (filterSuffix.isEmpty() || filterSuffix.contains(suffix)) {
                        if (!uncompressSizeMap.containsKey(suffix)) {
                            uncompressSizeMap.put(suffix, size.getFirst());
                        } else {
                            uncompressSizeMap.put(suffix, uncompressSizeMap.get(suffix) + size.getFirst());
                        }
                        if (!compressSizeMap.containsKey(suffix)) {
                            compressSizeMap.put(suffix, size.getSecond());
                        } else {
                            compressSizeMap.put(suffix, compressSizeMap.get(suffix) + size.getSecond());
                        }
                    } else {
//                        Log.d(TAG, "file: %s, filter by suffix.", entry.getKey());
                    }
                }
            }

            for (String suffix : uncompressSizeMap.keySet()) {
            //大小比对
                if (uncompressSizeMap.get(suffix).equals(compressSizeMap.get(suffix))) {
                    JsonObject fileItem = new JsonObject();
                    fileItem.addProperty("suffix", suffix);
                    fileItem.addProperty("total-size", uncompressSizeMap.get(suffix));
                    jsonArray.add(fileItem);
                }
            }
            ((TaskJsonResult) taskResult).add("files", jsonArray);
            taskResult.setStartTime(startTime);
            taskResult.setEndTime(System.currentTimeMillis());
            return taskResult;
        } catch (Exception e) {
            throw new TaskExecuteException(e.getMessage(), e);
        }
    }

CountRTask

目的：统计R文件数量。

  @Override
    public TaskResult call() throws TaskExecuteException {
        try {
            TaskResult taskResult = TaskResultFactory.factory(type, TaskResultFactory.TASK_RESULT_TYPE_JSON, config);
            long startTime = System.currentTimeMillis();
            Map<String, String> classProguardMap = config.getProguardClassMap();
            for (RandomAccessFile dexFile : dexFileList) {
                DexData dexData = new DexData(dexFile);
                dexData.load();
                ClassRef[] defClassRefs = dexData.getInternalReferences();
                for (ClassRef classRef : defClassRefs) {
                    String className = ApkUtil.getNormalClassName(classRef.getName());
                    if (classProguardMap.containsKey(className)) {
                        className = classProguardMap.get(className);
                    }
                    //去掉内部类
                    String pureClassName = getOuterClassName(className);
                    //识别R文件
                    if (pureClassName.endsWith(".R") || "R".equals(pureClassName)) {
                        if (!classesMap.containsKey(pureClassName)) {
                            classesMap.put(pureClassName, classRef.getFieldArray().length);
                        } else {
                            classesMap.put(pureClassName, classesMap.get(pureClassName) + classRef.getFieldArray().length);
                        }
                    }
                }
            }

            JsonArray jsonArray = new JsonArray();
            long totalSize = 0;
            Map<String, String> proguardClassMap = config.getProguardClassMap();
            for (Map.Entry<String, Integer> entry : classesMap.entrySet()) {
                JsonObject jsonObject = new JsonObject();
                if (proguardClassMap.containsKey(entry.getKey())) {
                    jsonObject.addProperty("name", proguardClassMap.get(entry.getKey()));
                } else {
                    jsonObject.addProperty("name", entry.getKey());
                }
                jsonObject.addProperty("field-count", entry.getValue());
                totalSize += entry.getValue();
                jsonArray.add(jsonObject);
            }
            ((TaskJsonResult) taskResult).add("R-count", jsonArray.size());
            ((TaskJsonResult) taskResult).add("Field-counts", totalSize);

            ((TaskJsonResult) taskResult).add("R-classes", jsonArray);
            taskResult.setStartTime(startTime);
            taskResult.setEndTime(System.currentTimeMillis());
            return taskResult;
        } catch (Exception e) {
            throw new TaskExecuteException(e.getMessage(), e);
        }
    }

DuplicateFileTask

目的：通过计算md5，判断apk中是否存在完全一样的文件。

private void computeMD5(File file) throws NoSuchAlgorithmException, IOException {
        if (file != null) {
            if (file.isDirectory()) {
                File[] files = file.listFiles();
                for (File resFile : files) {
                    computeMD5(resFile);
                }
            } else {
                MessageDigest msgDigest = MessageDigest.getInstance("MD5");
                BufferedInputStream inputStream = new BufferedInputStream(new FileInputStream(file));
                byte[] buffer = new byte[512];
                int readSize = 0;
                long totalRead = 0;
                while ((readSize = inputStream.read(buffer)) > 0) {
                    msgDigest.update(buffer, 0, readSize);
                    totalRead += readSize;
                }
                inputStream.close();
                if (totalRead > 0) {
                    final String md5 = Util.byteArrayToHex(msgDigest.digest());
                    String filename = file.getAbsolutePath().substring(inputFile.getAbsolutePath().length() + 1);
                    if (entryNameMap.containsKey(filename)) {
                        filename = entryNameMap.get(filename);
                    }
                    if (!md5Map.containsKey(md5)) {
                        md5Map.put(md5, new ArrayList<String>());
                        if (entrySizeMap.containsKey(filename)) {
                            fileSizeList.add(Pair.of(md5, entrySizeMap.get(filename).getFirst()));
                        } else {
                            fileSizeList.add(Pair.of(md5, totalRead));
                        }
                    }
                    //md5相同的文件列表
                    md5Map.get(md5).add(filename);
                }
            }
        }
    }

@Override
    public TaskResult call() throws TaskExecuteException {
            ...
            ...
            for (Pair<String, Long> entry : fileSizeList) {
                //md5相同的文件
                if (md5Map.get(entry.getFirst()).size() > 1) {
                    JsonObject jsonObject = new JsonObject();
                    jsonObject.addProperty("md5", entry.getFirst());
                    jsonObject.addProperty("size", entry.getSecond());
                    JsonArray jsonFiles = new JsonArray();
                    for (String filename : md5Map.get(entry.getFirst())) {
                        jsonFiles.add(filename);
                    }
                    jsonObject.add("files", jsonFiles);
                    jsonArray.add(jsonObject);
                }
            }
            ((TaskJsonResult) taskResult).add("files", jsonArray);
            taskResult.setStartTime(startTime);
            taskResult.setEndTime(System.currentTimeMillis());
        } catch (Exception e) {
            throw new TaskExecuteException(e.getMessage(), e);
        }
        return taskResult;
    }

MultiSTLCheckTask

目的：判断so是否带有多份stl标准库。

@Override
    public TaskResult call() throws TaskExecuteException {
        try {
            ...
            for (File libFile : libFiles) {
                if (isStlLinked(libFile)) {
                    Log.d(TAG, "lib: %s has stl link", libFile.getName());

                    jsonArray.add(libFile.getName());
                }
            }
            ((TaskJsonResult) taskResult).add("stl-lib", jsonArray);
            if (jsonArray.size() > 1) {
                ((TaskJsonResult) taskResult).add("multi-stl", true);
            } else {
                ((TaskJsonResult) taskResult).add("multi-stl", false);
            }
            taskResult.setStartTime(startTime);
            taskResult.setEndTime(System.currentTimeMillis());
            return taskResult;
        } catch (Exception e) {
            throw new TaskExecuteException(e.getMessage(), e);
        }
    }

private boolean isStlLinked(File libFile) throws IOException, InterruptedException {
        ProcessBuilder processBuilder = new ProcessBuilder(toolnmPath, "-D", "-C", libFile.getAbsolutePath());
        Process process = processBuilder.start();
        BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
        String line = reader.readLine();
        while (line != null) {
            String[] columns = line.split(" ");
//            Log.d(TAG, "%s", line);
            if (columns.length >= 3 && columns[1].equals("T") && columns[2].startsWith("std::")) {
                return true;
            }
            line = reader.readLine();
        }
        reader.close();
        process.waitFor();
        return false;
    }

UnusedResourcesTask

目的：检测出在代码、资源文件中未被引用的资源。

 @Override
    public TaskResult call() throws TaskExecuteException {
        try {
            TaskResult taskResult = TaskResultFactory.factory(type, TaskResultFactory.TASK_RESULT_TYPE_JSON, config);
            long startTime = System.currentTimeMillis();
            readMappingTxtFile();
            readResourceTxtFile();
            //添加所有声明的资源
            unusedResSet.addAll(resourceDefMap.values());
            Log.d(TAG, "find resource declarations %d items.", unusedResSet.size());
            //找到所有代码中使用的资源
            decodeCode();
            Log.d(TAG, "find resource references in classes: %d items.", resourceRefSet.size());
            //找到所有资源中引用的资源
            decodeResources();
            Log.d(TAG, "find resource references %d items.", resourceRefSet.size());
            //去掉被引用的资源
            unusedResSet.removeAll(resourceRefSet);
            Log.d(TAG, "find unused references %d items", unusedResSet.size());

            JsonArray jsonArray = new JsonArray();
            for (String name : unusedResSet) {
                jsonArray.add(name);
            }
            ((TaskJsonResult) taskResult).add("unused-resources", jsonArray);
            taskResult.setStartTime(startTime);
            taskResult.setEndTime(System.currentTimeMillis());
            return taskResult;
        } catch (Exception e) {
            throw new TaskExecuteException(e.getMessage(), e);
        }
    }

private void readMappingTxtFile() throws IOException {
        // com.tencent.mm.R$string -> com.tencent.mm.R$l:
        //      int fade_in_property_anim -> aRW

        if (mappingTxt != null) {
            BufferedReader bufferedReader = new BufferedReader(new FileReader(mappingTxt));
            String line = bufferedReader.readLine();
            boolean readRField = false;
            String beforeClass = "", afterClass = "";
            try {
                while (line != null) {
                    if (!line.startsWith(" ")) {
                        String[] pair = line.split("->");
                        if (pair.length == 2) {
                            beforeClass = pair[0].trim();
                            afterClass = pair[1].trim();
                            afterClass = afterClass.substring(0, afterClass.length() - 1);
                            if (!Util.isNullOrNil(beforeClass) && !Util.isNullOrNil(afterClass) && ApkUtil.isRClassName(ApkUtil.getPureClassName(beforeClass))) {
//                                Log.d(TAG, "before:%s,after:%s", beforeClass, afterClass);
                                readRField = true;
                            } else {
                                readRField = false;
                            }
                        } else {
                            readRField = false;
                        }
                    } else {
                        if (readRField) {
                            String[] entry = line.split("->");
                            if (entry.length == 2) {
                                String key = entry[0].trim();
                                String value = entry[1].trim();
                                if (!Util.isNullOrNil(key) && !Util.isNullOrNil(value)) {
                                    String[] field = key.split(" ");
                                    if (field.length == 2) {
//                                        Log.d(TAG, "%s -> %s", afterClass.replace('$', '.') + "." + value, getPureClassName(beforeClass).replace('$', '.') + "." + field[1]);
                                        //添加 R.java中混淆后的全路径field -> R.java混淆前的全路径field
                                        rclassProguardMap.put(afterClass.replace('$', '.') + "." + value, ApkUtil.getPureClassName(beforeClass).replace('$', '.') + "." + field[1]);
                                    }
                                }
                            }
                        }
                    }
                    line = bufferedReader.readLine();
                }
            } finally {
                bufferedReader.close();
            }
        }
    }

private void readResourceTxtFile() throws IOException {
        //读取R.txt
        BufferedReader bufferedReader = new BufferedReader(new FileReader(resourceTxt));
        String line = bufferedReader.readLine();
        try {
            while (line != null) {
                String[] columns = line.split(" ");
                if (columns.length >= 4) {
                    final String resourceName = "R." + columns[1] + "." + columns[2];
                    if (!columns[0].endsWith("[]") && columns[3].startsWith("0x")) {
                        //int styleable ActionBar_title 27
                        if (columns[3].startsWith("0x01")) {
                            Log.d(TAG, "ignore system resource %s", resourceName);
                        } else {
                            final String resId = parseResourceId(columns[3]);
                            if (!Util.isNullOrNil(resId)) {
                                //资源id 资源名称 映射
                                resourceDefMap.put(resId, resourceName);
                            }
                        }
                    } else {
                        //int[] styleable ActionMode { 0x7f030034, 0x7f030036, 0x7f030056, 0x7f0300ad, 0x7f030168, 0x7f03019e }
                        Log.d(TAG, "ignore resource %s", resourceName);
                        if (columns[0].endsWith("[]") && columns.length > 5) {
                            Set<String> attrReferences = new HashSet<String>();
                            for (int i = 4; i < columns.length; i++) {
                                if (columns[i].endsWith(",")) {
                                    attrReferences.add(columns[i].substring(0, columns[i].length() - 1));
                                } else {
                                    attrReferences.add(columns[i]);
                                }
                            }
                            //style映射
                            styleableMap.put(resourceName, attrReferences);
                        }
                    }
                }
                line = bufferedReader.readLine();
            }
        } finally {
            bufferedReader.close();
        }
    }

解析dex文件中的smali代码：

private void decodeCode() throws IOException {
        for (String dexFileName : dexFileNameList) {
            DexBackedDexFile dexFile = DexFileFactory.loadDexFile(new File(inputFile, dexFileName), Opcodes.forApi(15));

            BaksmaliOptions options = new BaksmaliOptions();
            List<? extends ClassDef> classDefs = Ordering.natural().sortedCopy(dexFile.getClasses());

            for (ClassDef classDef : classDefs) {
                String[] lines = ApkUtil.disassembleClass(classDef, options);
                if (lines != null) {
                    readSmaliLines(lines);
                }
            }

        }
    }

private void readSmaliLines(String[] lines) {
        if (lines == null) {
            return;
        }
        for (String line : lines) {
            line = line.trim();
            if (!Util.isNullOrNil(line)) {
                if (line.startsWith("const")) {
                    String[] columns = line.split(",");
                    if (columns.length == 2) {
                        final String resId = parseResourceId(columns[1].trim());
                        //从id获取资源名
                        if (!Util.isNullOrNil(resId) && resourceDefMap.containsKey(resId)) {
                            resourceRefSet.add(resourceDefMap.get(resId));
                        }
                    }
                } else if (line.startsWith("sget")) {
                    String[] columns = line.split(" ");
                    if (columns.length == 3) {
                        //获取资源名称
                        final String resourceRef = parseResourceNameFromProguard(columns[2]);
                        if (!Util.isNullOrNil(resourceRef)) {
                            //Log.d(TAG, "find resource reference %s", resourceRef);
                            if (styleableMap.containsKey(resourceRef)) {
                                //reference of R.styleable.XXX
                                for (String attr : styleableMap.get(resourceRef)) {
                                    resourceRefSet.add(resourceDefMap.get(attr));
                                }
                            } else {
                                resourceRefSet.add(resourceRef);
                            }
                        }
                    }
                }
            }
        }
    }

private String parseResourceNameFromProguard(String entry) {
        if (!Util.isNullOrNil(entry)) {
            // sget v6, Lcom/tencent/mm/R$string;->chatting_long_click_menu_revoke_msg:I
            // sget v1, Lcom/tencent/mm/libmmui/R$id;->property_anim:I
            // sput-object v0, Lcom/tencent/mm/plugin_welab_api/R$styleable;->ActionBar:[I
            // const v6, 0x7f0c0061
            String[] columns = entry.split("->");
            if (columns.length == 2) {
                int index = columns[1].indexOf(':');
                if (index >= 0) {
                    final String className = ApkUtil.getNormalClassName(columns[0]);
                    final String fieldName = columns[1].substring(0, index);
                    if (!rclassProguardMap.isEmpty()) {
                        String resource = className.replace('$', '.') + "." + fieldName;
                        if (rclassProguardMap.containsKey(resource)) {
                            return rclassProguardMap.get(resource);
                        } else {
                            return "";
                        }
                    } else {
                        if (ApkUtil.isRClassName(ApkUtil.getPureClassName(className))) {
                            return (ApkUtil.getPureClassName(className) + "." + fieldName).replace('$', '.');
                        }
                    }
                }
            }
        }
        return "";
    }

UnusedAssetsTask

目的：检测出apk中未被使用的asset资源（代码实现仅覆盖了字符串常量的情况，会有遗留）。

 @Override
    public TaskResult call() throws TaskExecuteException {
        try {
            TaskResult taskResult = TaskResultFactory.factory(type, TaskResultFactory.TASK_RESULT_TYPE_JSON, config);
            long startTime = System.currentTimeMillis();
            File assetDir = new File(inputFile, ApkConstants.ASSETS_DIR_NAME);
            //找到所有asset文件
            findAssetsFile(assetDir);
            generateAssetsSet(assetDir.getAbsolutePath());
            Log.d(TAG, "find all assets count: %d", assetsPathSet.size());
            //解析代码中的asset引用
            decodeCode();
            Log.d(TAG, "find reference assets count: %d", assetRefSet.size());
            //移除被引用的资源
            assetsPathSet.removeAll(assetRefSet);
            JsonArray jsonArray = new JsonArray();
            for (String name : assetsPathSet) {
                jsonArray.add(name);
            }
            ((TaskJsonResult) taskResult).add("unused-assets", jsonArray);
            taskResult.setStartTime(startTime);
            taskResult.setEndTime(System.currentTimeMillis());
            return taskResult;
        } catch (Exception e) {
            throw new TaskExecuteException(e.getMessage(), e);
        }
    }

private void generateAssetsSet(String rootPath) {
        HashSet<String> relativeAssetsSet = new HashSet<String>();
        for (String path : assetsPathSet) {
            int index = path.indexOf(rootPath);
            if (index >= 0) {
                String relativePath = path.substring(index + rootPath.length() + 1);
                //Log.d(TAG, "assets %s", relativePath);
                relativeAssetsSet.add(relativePath);
                if (ignoreAsset(relativePath)) {
                    Log.d(TAG, "ignore assets %s", relativePath);
                    //获取asset使用时的相对路径
                    assetRefSet.add(relativePath);
                }
            }
        }
        assetsPathSet.clear();
        assetsPathSet.addAll(relativeAssetsSet);
    }

private void readSmaliLines(String[] lines) {
        if (lines == null) {
            return;
        }
        for (String line : lines) {
            line = line.trim();
            //    invoke-virtual {p0}, Lcom/ss/android/alog/App;->getAssets()Landroid/content/res/AssetManager;

            //move-result-object v1

            //const-string v2, "video"

            //invoke-virtual {v1, v2}, Landroid/content/res/AssetManager;->open(Ljava/lang/String;)Ljava/io/InputStream;
            //:try_end_13
            //.catch Ljava/io/IOException; {:try_start_a .. :try_end_13} :catch_1a
            //这个const-string判断不是很完善，只能判断写死的值
            if (!Util.isNullOrNil(line) && line.startsWith("const-string")) {
                String[] columns = line.split(",");
                if (columns.length == 2) {
                    String assetFileName = columns[1].trim();
                    assetFileName = assetFileName.substring(1, assetFileName.length() - 1);
                    if (!Util.isNullOrNil(assetFileName)) {
                        //再判断这个常量是否在asset文件名集合中
                        for (String path : assetsPathSet) {
                            if (path.endsWith(assetFileName)) {
                                assetRefSet.add(path);
                            }
                        }
                    }
                }
            }
        }
    }

UnStrippedSoCheckTask

目的：检测出apk中未裁剪的so。

@Override
    public TaskResult call() throws TaskExecuteException {
        try {
            ...
            if (libDir.exists() && libDir.isDirectory()) {
                File[] dirs = libDir.listFiles();
                for (File dir : dirs) {
                    if (dir.isDirectory()) {
                        File[] libs = dir.listFiles();
                        for (File libFile : libs) {
                            if (libFile.isFile() && libFile.getName().endsWith(ApkConstants.DYNAMIC_LIB_FILE_SUFFIX)) {
                                libFiles.add(libFile);
                            }
                        }
                    }
                }
            }
            for (File libFile : libFiles) {
                //判断是否裁剪
                if (!isSoStripped(libFile)) {
                    Log.d(TAG, "lib: %s is not stripped", libFile.getName());

                    jsonArray.add(libFile.getName());
                }
            }
            ((TaskJsonResult) taskResult).add("unstripped-lib", jsonArray);
            taskResult.setStartTime(startTime);
            taskResult.setEndTime(System.currentTimeMillis());
            return taskResult;
        } catch (Exception e) {
            throw new TaskExecuteException(e.getMessage(), e);
        }
    }

通过命令行判断so是否被裁剪

private boolean isSoStripped(File libFile) throws IOException, InterruptedException {
        ProcessBuilder processBuilder = new ProcessBuilder(toolnmPath, libFile.getAbsolutePath());
        Process process = processBuilder.start();
        BufferedReader reader = new BufferedReader(new InputStreamReader(process.getErrorStream()));
        String line = reader.readLine();
        if (!Util.isNullOrNil(line)) {
            //Log.d(TAG, "%s", line);
            String[] columns = line.split(":");
            if (columns.length == 3 && columns[2].trim().equalsIgnoreCase("no symbols")) {
                return true;
            }
        }
        reader.close();
        process.waitFor();
        return false;
    }

CountClassTask

目的：统计类的数量。

@Override
    public TaskResult call() throws TaskExecuteException {
        try {
            TaskResult taskResult = TaskResultFactory.factory(type, TaskResultFactory.TASK_RESULT_TYPE_JSON, config);
            long startTime = System.currentTimeMillis();
            Map<String, String> classProguardMap = config.getProguardClassMap();
            JsonArray dexFiles = new JsonArray();

            for (int i = 0; i < dexFileList.size(); i++) {
                RandomAccessFile dexFile = dexFileList.get(i);
                DexData dexData = new DexData(dexFile);
                dexData.load();
                ClassRef[] defClassRefs = dexData.getInternalReferences();
                Set<String> classNameSet = new HashSet<>();
                for (ClassRef classRef : defClassRefs) {
                    String className = ApkUtil.getNormalClassName(classRef.getName());
                    if (classProguardMap.containsKey(className)) {
                        className = classProguardMap.get(className);
                    }
                    if (className.indexOf('.') == -1) {
                        continue;
                    }
                    classNameSet.add(className);
                }
                JsonObject jsonObject = new JsonObject();
                jsonObject.addProperty("dex-file", dexFileNameList.get(i));
                //Log.d(TAG, "dex %s, classes %s", dexFileNameList.get(i), classNameSet.toString());

                Map<String, Set<String>> packageClass = new HashMap<>();
                if (JobConstants.GROUP_PACKAGE.equals(group)) {
                    String packageName = "";
                    for (String clazzName : classNameSet) {
                        packageName = ApkUtil.getPackageName(clazzName);
                        if (!Util.isNullOrNil(packageName)) {
                            if (!packageClass.containsKey(packageName)) {
                                packageClass.put(packageName, new HashSet<String>());
                            }
                            //按package聚合
                            packageClass.get(packageName).add(clazzName);
                        }
                    }
                    JsonArray packages = new JsonArray();
                    for (Map.Entry<String, Set<String>> pkg : packageClass.entrySet()) {
                        JsonObject pkgObj = new JsonObject();
                        pkgObj.addProperty("package", pkg.getKey());
                        JsonArray classArray = new JsonArray();
                        for (String clazz : pkg.getValue()) {
                            classArray.add(clazz);
                        }
                        //单个package下的所有class
                        pkgObj.add("classes", classArray);
                        packages.add(pkgObj);
                    }
                    jsonObject.add("packages", packages);
                }
                dexFiles.add(jsonObject);
            }

            ((TaskJsonResult) taskResult).add("dex-files", dexFiles);
            taskResult.setStartTime(startTime);
            taskResult.setEndTime(System.currentTimeMillis());
            return taskResult;
        } catch (Exception e) {
            throw new TaskExecuteException(e.getMessage(), e);
        }
    }

总结

Matrix静态apk扫描部分的代码逻辑比较简单；初步理解dex文件格式、arsc文件格式之后，代码理解上就不会有太大的问题了。